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A TWO-SAMPLE TEST 


Lincoun E. Moses 


TEACHERS COLLEGE, COLUMBIA UNIVERSITY 


In certain investigations involving an experimental group and a control 
group, the effect of the experimental treatment may be expected to manifest 
itself by increasing the scores of some subjects but by decreasing the scores of 
other subjects. Examples arise in the area of defensive behavior. Customary 
tests of differences between means are inappropriate for assessing the existence 
of effect of the experimental treatment in such cases. A test based on the 
ranks of the observations is proposed; it will be sensitive to extreme scores 
(either large or small or both) for experimentals. 


Often in psychological studies where behavior can be represented by 
scores along a continuum, the import of extremely large or extremely small 
scores is much the same. For instance, in certain association or recognition 
situations either a rapid or a delayed response is evidence of defensive be- 
havior. Similar considerations apply for many “personality” variables. For 
convenience of exposition we shall employ terminology special to perceptual 
defense. 

Suppose that an investigator wishes to determine whether the behavior 
in one group (experimentals) is defensive, as contrasted with the behavior 
of another group (controls). He then faces the difficulty that statistical 
techniques based upon differences of means (or medians, or mean ranks, etc.) 
may be inherently invalid for the problem. Suppose that on each subject 
is obtained one measurement which, if small, indicates a defensive response 
sometimes characterized as “‘vigilant.”” If the number is large it represents a 
defensive response of a character sometimes called ‘“‘repressive.’”? The normal 
response is expected not to be extreme in either direction. Now if the experi- 
mental group consists of subjects, some of whom give “vigilant” defensive 
responses and some “repressive” defensive responses, their mean response 
may be quite close to the mean response of the controls. The null hypothesis 
might well be accepted when it should be rejected. 

Since a test of means is not in order, the experimenter might propose to 
use the Wald-Wolfowitz run test. He would not be much better off than he 
was before, since the run test will merely tell him whether or not the two sets 
of measurements can be viewed as coming from a common population. If the 
result is that they can not, he still has no firm basis for stating that the experi- 
mentals exhibited defense as compared with the controls. For instance, 
suppose that all scores are arranged in order of increasing size and that one- 
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half of the control group scores lay at the beginning and were followed by 
all the experimental group scores which in turn were followed by the remaining 
control group scores. The run test here would reject the hypothesis of a 
common population but the character of the sample would certainly not 
justify the statement that the experimentals had given defensive responses; 
rather it would indicate that the controls had. 

It would seem, then, that a suitable test is needed. Proposed in this 
paper is a test of the null hypothesis that the two groups, experimental and 
control, come from a common population, against the alternative that the 
experimentals give defensive responses. (More generally the alternative hy- 
pothesis is that the experimentals are “extreme” in one or both directions, 
relative to the controls.) The test is completely distribution-free. It will 
have its specified significance level whatever be the form of the common 
population, when the null hypothesis is true. It is not argued that this is the 
best test of the hypothesis which can be given, in the sense that presumably 
more sensitive ones could be constructed. The test has the desirable property 
that there is not an implicit assumption of a measurement scale with equal 
intervals. 

It should be emphasized that the test to be proposed is specifically 
designed for the case where the best information of the investigator indicates 
that the experimental condition may affect the scores of some (or possibly all) 
subjects in one way and the scores of some (or possibly all) subjects in the 
opposite way. In this case the customary tests of location (or “slippage’’) 
based on means or medians or rank sums cannot be relied upon to give reason- 
able answers even with very large samples. However, if there exist a priori 
grounds for expecting that the effect (if any) of the experimental condition 
must be in the same direction for all subjects affected, then a suitable test 
of location is not only appropriate, but is to be preferred to the test here 
proposed. 

For definiteness, let there be m observations from the control group which 
we will call z’s and let there be n observations from the experimental group 
which we will call y’s. Let the m + n observations be arranged in order of 
increasing size. If the null hypothesis that the x’s and y’s come from a common 
population is true, we should expect the z’s and y’s to be well mixed in this 
ordered arrangement. For instance, some of the largest observations should 
be z’s, and some y’s; some of the smallest observations should be z’s, and some 
y's. However, if the alternative hypothesis that the y’s are defensive re- 
sponses is true, then we should expect to find either most of the y’s being 
small observations, in which case they would be at the left end of the scale, 
or most of the y’s being large observations in which case they would be at the 
right end of the scale, or a sizable number of y’s at one end and a sizable 
number at the other end (some of the defensive responses being “vigilant,” 
others being “repressive’’). In any of these three situations the control 
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group scores will be unduly congested, at one end, at the other end, or in the 
middle. The character of the test, then, is to determine whether the 2’s 
(control group scores) are so closely compacted as to call for rejecting the 
null hypothesis in favor of the alternative hypothesis. 

The construction of a suitable test depends, then, on doing two things: 
1. Finding a satisfactory statistic, s*, to measure the degree to which the z’s 
are compacted. 2. Finding a number c such that if the null hypothesis is 
true the probability that s* should be less than or equal to c is small, say 
.05 or .01. 

The choice of the statistic s* to measure the degree of compactedness of 
the control group scores must be made in the light of the second requirement; 
that is, we must be able to find its distribution under the null hypothesis. 
Accordingly we might take our measure of compactedness to be the range of 
values occupied by the group of 2’s in the ordered arrangement of all z’s 
and y’s. 

Consider the following hypothetical scores: 


x (controls) y (experimentals) 
aed 2.7 1.3 3.5 
1.8 2.8 1.5 3.8 
1.9 3.1 1.6 4.5 
2.2 3.6 3.4 6.7 

3.7 


We arrange these in order of increasing size and obtain the array: 


Le) 460 2:6 2.0 BB 159) 222i QS Sek S44. SS SOS SB) 4.51 657 
y y y z x x z x x z y y x x y y y 
1 2 3 4 5 6 7 S. 19F TOP “TE Ser fot F416: 10%) he 


In this array the obtained scores are given in the first row, their identifications 
as x or y in the second row, and their ranks (from least to greatest) in the third 
row. The smallest xz has rank 4, the largest x has rank 14. The total “span” 
of the z’s is over 11 ranks; that is, in order to include all the z’s we must take 
a consecutive block of 11 ranks, 4 through 14 inclusive. We shall define our 
measure of compactedness to be the smallest number of consecutive ranks 
necessary to include all the z’s; we shall use the symbol s* to denote this 
statistic. Observe that s* is equal to one plus the difference between the 
extreme x ranks. For example, s* = 11, m = 9,n = 8. It would remain to 
determine whether 11 is too small a value for s* reasonably to arise by chance 
if the z’s and y’s are from a common population. 

A defect of taking the measure of compactedness to be s*, as defined, is 
that it depends so strongly on the two extreme scores of the control group. 
This is open to objections; first, if the control group be contaminated with a 
few members who might more properly be classified as experimentals, often 
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when we should reject the null hypothesis, we will fail to do so merely because 
these few members of the control group will by themselves determine the 
value of the statistic s*; second, where m is large, the range (obviously about 
the same thing as “‘span’’) is an inefficient index of the compactedness or 
spread of the group of 2’s. 

Both objections can be met in part by considering modifications of s*. 
We might instead of s* consider the span of the middle m — 2h 2’s where h 
is arbitrarily selected as some fairly small number. That is, we find the span 
of the m 2’s after the extreme h x’s at each end are omitted from consideration. 
For instance, in the example if we should decide to take h = 2, then m — 2h = 
9—4=5. These 5 middle 2’s occupy ranks 6 through 10 and this ‘truncated 
span” is 5. We can denote this by the statement 


s% = 5, h = 2. 


So, we define s*, as the least number of consecutive ranks necessary to include 
all the x’s, except for the h least and the A greatest of them. That is, the h 
least and A largest x’s are omitted from consideration, and s* is the span of 


the remaining m — 2h z’s. 

If we adopt s% as our test statistic we then have the task of finding the 
probabilities associated with small values of s* . Since s% can never be less 
than m — 2h, and can never be greater than n + m — 2h, the problem is to 
determine how much larger than m — 2h it can become and still remain 
convincingly small. What is needed, then, is to find the largest possible integer 
r, such that for a predetermined level of significance, a, 


P{si < m— 2h+r} <a. (1) 


When the largest such value of r is found then we have the criterion: Reject 
the null hypothesis if 


si <m—2h+r (2) 


and otherwise accept it. The distribution theory of s% is worked out in the 
brief mathematical appendix. The result is given in formula (3), which states 
that where there are m x’s and n y’s and s¥, is defined as above, the probability 
that s% < m — 2h + r is equal to the sum of r + 1 terms, each a product of 
certain combinatorials, divided by the total possible combinations of m + n 
objects taken m at a time. 

> CaeaP atlas ‘age eames 

a n—1 


P{st < m— 2h+r} = (m 7 5 , 


m 





(3) 


where for any positive integers, a and b, the symbol (*) represents the 


combinations of a things taken b at a time; that is, 
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a a! 5 
= ——_ if a> b, 
(?) (a — b)!b! (4) 


= 0 if eo < %. 


Then for any m, n, and h one can find the largest value of s% which is signifi- 
cant at some significance level, say .05, by cumulating the terms in the numer- 
ator until reaching the largest value of r for which the quotient is smaller 
than .05. A table of significant values of s* for various m, n, h is given; for 
other values of m and n or h the investigator may find significant values by 
using formula (3). 

For any choice of h between zero and (m — 1)/2 there is a test based on 
the corresponding s*%. Only one such test should be applied to any one set 
of data. Which h to choose is a matter not too easily decided. Any choice 
will be legitimate so long as it is made before the observations are taken. 
In general as m increases, h should increase. The tables given in this paper 
have taken h = m/4. A relatively large value of h such as this protects the 
experimenter if the control group is contaminated with subjects who should 
better have been in the experimental group. A small value of h, such as say 
h = m/15 or h = O, protects the experimenter if the experimental group is 
contaminated with subjects who should better have been in the control group. 


Example: 


Suppose that there are 16 in the control group (m = 16) and 12 in the 
experimental group (n = 12) and that the scores are: 


z: 21.0, 21.4, 22.5, 23.6, 25.8, 25.9, 27.0, 27.8 
30.1, 30.5, 33.7, 34.8, 35.1, 35.3, 38.9, 45.0 


y: 17.4, 18.6, 19.1, 22.3, 24.7, 25.2 
26.1, 27.1, 28.0, 29.5, 30.3, 30.4 


Arrangement of these scores from least to greatest yields the following se- 
quence of x’s and y’s, together with their ranks. 


gy eS ey Tea ay 2s wy Se eae as Se yk Oe Se Ps se 
123456789 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 


In this array the values of s*% for certain values of h are 
h=0;s§ = 28-— 4+1 = 25 
h=1;s§ = 27-— 5+1= 23 


h=2;s§ = 26-— 7+1= 20 (5) 
h=3;s%§ = 25- 8+1= 18 
h=4;s% = 24-—11+1= 14 


For m = 16, n = 12, and h = 4 we find from Table 2 that s% as small as 
10 is significant at level .057. Since our value 14 is not this small we find 
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no evidence of defensive behavior in the experimental group, as compared 
with the control group. 

Had the experimenter, in advance of taking the observations, decided to 
use h = 1, then the value of his test statistic would be 23 and he would have 
to find the significance level for the statistic by using formula (3). In doing 


TABLE 1 
Probability That ss <m+r-—2h 
When m = 12, h = 3, for Various Values of n 

















m+r—2h 
n 
6 7 8 9 
6 .092 
8 .051 
12 .019 .077 
18 .006 .026 .068 
24 .002 .010 .029 .062 
TABLE 2 


Probability That 4 <m-+r-—2h 
When m = 16, h = 4, for Various Values of n 























m+r— 2h 

n 
8 9 10 11 12 13 14 
8 .033 142 
12 .010 .048 . 133 
16 .003 .019 .057 
24 .001 .004 .013 .032 .065 
32 .000 .001 .004 .010 .021 .041 .070 
TABLE 3 
Probability That 8; <_m +r — 2h 
When m = 20, A = 5, for Various Values of n 

m+r-— 2h 

n 





10 11 12 13 14 15 16 17 18 19 





10 .012 .062 

15 002 .015 .049 .118 

20 001 .004 .015 .041 .088 

30 000 .001 .002 .006 .015 .031 # .058 

40 000 .000 .000 .001 003 6.007) «3=.015)=— «.026-—s—(«i«w44ASt«i«w GD 
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TABLE 4 
Probability That st <m +r — 2h 
When m = 32, h = 8, for Various Values of n 








m+r— 2h 





16 17 18 19 20 21 22 23 24 25 26 27 28 #29 #30 31 32 33 34 





16 =.001 .004 .018 .055 

24 .000 .000 .002 .006 .017 .040 .081 

32 .000 .000 .000 .001 .002 .007 .016 .032 .059 

48  .000 .000 .000 .000 .000 .000 .001 .002 .004 .008 .014 .023 .037 .058 

64 .000 .000 .000 .000 .000 .000 .000 .000 .000 .001 .001 .003 .005 .008 .012 .019 .028 .040 .056 





this computation the work would be greatly facilitated by use of a table of 
logarithms of factorials. 

In case there are ties in the ranks the value of s{ may be different for 
different ways of breaking the ties. When this is true break the ties in all 
possible ways, find the probability level associated with the sample for each 
such way, and average these probabilities. 


Appendix 


Where there are m z’s and n y’s arranged in order of ascending size 
define s*% as the least number of consecutive ranks necessary to include all 
x’s except the first h of them and the last h of them. 

Then 

m— 2h<s&t < m+n — 2h. (6) 


The problem is to find P{s% = g}, m — 2h< gs m+n — 2h. 

To find the probability that s% = q we first find the number of ways in 
which we can have s% = gq, and then divide by the total number of ways in 
which we can fill the m + n rank positions with m 2’s and n y’s. 

To have s% = q we must have h 2’s in the first 7 observations, then an z, 
then m — 2h — 2 2’s in the next g — 2 observations, then an z, then h z’s 
in the last m + n — j — q observations. Hereh <j < m+n—q-—Ah. 
Diagrammatically this statement is represented in Figure 1. 

Here the least x (except for the h smallest of them) occurs at the left- 
hand encircled rank. To the left there are j smaller observations, of which 
exactly h are x’s. The right-hand encircled rank represents the largest x 
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(except for the largest h of them). Between the two encircled ranks lie g — 2 
observations, among which are all the z’s except for the h largest, h smallest, 
and the two 2’s at the ends of span s* ; that is, there are m — 2h — 2 2’s 
among these g — 2 observations. The remaining m + n — gq — j observations 
to the right of the right-hand encircled rank contain exactly h x’s. Then for 
any fixed j between h and n + A, inclusive, the number of ways in which 


we can have s% = gq is 


TO aa 
h/\m — 2h — 2 h i 


And to find the total number of ways in which we can have s% = q we must 
sum this over the range of j, giving as the number of ways for s% = q, the 


quantity 
m+n—q—h * ‘ j 
J q-—2 et ae 
2, Xs —2h—-—2 h ’ (8) 
which may be written (by replacing 7 by k = 7 — h) as: 


q—2 -_ Too Se 
ie. — 2h-—2 dX h h : (9) 


To evaluate this we use an identity of probability theory,* 


ee age ee eee (10) 


I= J K—-J K 
Taking h + 1 for B, m + n — 2h — q for K, h for A — 1, and k for J, we 
obtain 
( q-2 core ers) (11) 
m — 2h — 2/\m +n — q — 2h] 


Now in order to find the number of ways in which we can have s% < m — 
2h + r, we must add the above expression for values of g less than or equal 
to m — 2h + r. Recalling that necessarily g > m — 2h, this gives the 


expression 
m—2h+r 
q-2 _ alg ipsa = 
2 Wea m+n— q-— 2h/’ - 


q=m—2h 
which may be written (by replacing g by 7 + m — 2h) as 


oa 
f<0\ m— 2h—2 n—1 
(13) 
es aoc eee Y 
i=} v a= % 
*Feller, W. An Introduction to Probability Theory and Its Applications. New York: 
John Wiley & Sons, 1950, p. 48. 
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Application of the identity earlier cited shows that when r = n, that is, we 
sum over all possible values of s% , the sum becomes i” "), as it should. 
Therefore, to write the probability that 7 < r or that g < m — 2h + 7, which 
is the same thing, we divide the above sum by e : and obtain 


aoe ees ~‘) 
P{st < m—2h+r} = = : (m . " — . (14) 
m 
And thus the correctness of formula (3) is demonstrated. It may be remarked 


that the distribution of s% is asymptotically normal, but the approximation is 
unsatisfactory for sample sizes of practical interest. 





Manuscript received 12/6/51 
Revised manuscript received 2/29/52 
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A NOTE ON CLUSTER-DIRECTED ANALYSIS 


C. J. Apcock 
VICTORIA UNIVERSITY COLLEGE 


Thurstone’s multiple group method of factor analysis has been widely 
used as a basis for rotation to simple structure. To make the most of the 
economy offered by this method, factor axes may be located directly by the 
correlation clusters; and methods of doing this are here discussed. 


Thurstone’s multiple group method (1) of factoring provides a simple 
and rapid means of analysis, but it depends upon rotation to simple structure 
to fix the reference axes. Since cluster selection can itself be used as a means 
to locate the reference axes (2, 3), it may be desirable to take advantage of the 
group method to avoid the tedious and sometimes uncertain task of rotation 
according to simple structural principles. This will involve more careful 
selection of clusters than is demanded in the usual application of the Thurstone 
group method; but, in most cases, the work will be considerably less than would 
be involved in the normal process of rotation. 

The objections to such a procedure as this are two. In the first place, 
factors based on clusters of the original correlations show considerable inter- 
correlation in most cases. For this reason it is usual to first remove the 
general factor either as a first centroid (2) or as a “‘basic”’ factor (3). In the 
second place, rotation to orthogonality is needed to check on the residual 
correlation. It should be noted, however, that oblique factors plus a second- 
order factor which accounts for this inter-correlation can be converted into 
equivalent orthogonal factors plus a general factor (4, p. 297) and this is 
what is achieved by Burt’s group method. Burt’s method, however, makes 
no provision for overlap, and he has to deal with such a situation by the 
somewhat drastic method of carrying out an ordinary centroid analysis in 
addition and then rotating to the group-factor positions. Actually, he carries 
out the centroid analysis first and bases his matrix sections on this instead of 
the cluster analysis. In practice this double analysis is much shorter than the 
usual rotation methods, as was demonstrated by Charlotte Banks (5); but 
the cluster-directed approach is still shorter and may have special advan- 
tages with some data. 

If we are prepared to accept orthogonal factors plus a general or basic 
factor instead of the usual oblique factors plus second-order factors (and on the 
grounds of simplicity a strong argument can be advanced in favor of this), 
we may meet both the above objections to cluster-based rotation of the 
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Thurstone’s group-factor matrix at once by an alternative method of rotating 
to orthogonality. Any orthogonal rotation will, of course, remove the inter- 
correlation but Thurstone’s method of basing on a diagonal analysis will not 
serve our purpose, since the first factor is used as a pivot and is so arbitrarily 
weighted. Subsequent factors are increasingly distorted and, if many factors 
are involved, the last factor may be forced far from the position suggested by 
its correlation cluster. 

We require an alternative method of rotation which will keep as closely 
as possible to the positions suggested by all the clusters. Under these cir- 
cumstances, provided we have made an adequate cluster search, we should 
find that one of our orthogonal factors corresponds to the basic factor of Burt, 
since it too must have a cluster effect. We can thus get a complete system of 
orthogonal factors by one simple rotation of the group-factor matrix. 

The alternative method of rotation to orthogonal position here referred 
to is one which requires that all the reference axes participate equally in the 
“give and take” required to attain orthogonality. This means that the trans- 
formation matrix, F;,,, must be symmetric. Our problem is thus to calculate 
from the interfactor correlation matrix, R,, , a factor matrix F,,, such that 
Fin = F},,.. Iam indebted to Dr. Ledyard Tucker for a precise solution to 
this problem. He takes as his basic equation, 


R,, = ADA’ (where A is orthogonal). (1) 


This is equation 110 from Thurstone’s text (1, p. 501). It applies to the 
principal-axes solution, the entries in the diagonal matrix D being the charac- 
teristic roots of R,,. It is required that 


Fyn = Fin - (2) 
If we take 
Fym = AD*A’, (3) 
which is clearly symmetric, then 
FymF'bm = AD*A’AD*A’ 
= AD'TD'a’ 
(4) 
= ADA’ 
= f.. by (1) as required 
and 
F5. = ADA’. (5) 


If we denote the factor matrix for the principal axes F,, , we get, in 


terms of Thurstone’s equation 105 (1, p. 501), 
A= F,,D™. (6) 











< \ ‘ 
Cc. J. ADCOCK 2515 j 


Substituting in (5) we get 
Fym = (FizD™)D“*(F,,D"')’ 


¢ pit AN 
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Post-multiplication of the original oblique factor matrix by F;, gives us 
the required orthogonal transformation, 


VF; = F. 


Once the principal-axes solution has been obtained, the transformation 
matrix is quickly obtained by equation (7). The chief objection to be urged 
against the method is the length of time required to find the principal axes, 
but where only a few factors are concerned this may not be too time-con- 
suming. For correlation tables involving high error variance, I have made 
use of an approximation method which involves considerably less work. Since 
much material hardly warrants laborious calculations and since the high error 
variance often results in considerable distortion when the usual simple-struc- 
ture methods of rotation are employed, it may be worth while to describe this 
simpler method. 

Briefly, the procedure is to obtain a transformation matrix as described 
by Thurstone and then to calculate a correction matrix which will rotate 
this to symmetric form (i.e., F;,, as above). The correction matrix should be 
orthogonal in order to keep the factor variance constant; and if the non- 
diagonal terms are chosen skew-symmetrically, the solution of the equations 
involved is facilitated. These two requirements are to some extent conflicting, 
but if the matrix is first calculated according to the latter requirement and 
then normalized, the compromise seems sufficiently close for the purpose here 
described. It should be noted that the requirement of skew symmetry is not 
exactly true of the precise solution. The skew symmetric terms can be 
calculated one at a time if we take the diagonal elements as unity. Denote 
the elements of the original transformation matrix ¢;; , the elements of the 
correction matrix c;; , and the elements of F;, as f;; . Now since F),, is 
symmetric, fos = fs2 ; so fora 3 X 3 matrix we can write 


tooCoz + tosC33 = bs3Cae - 
But c;; = unity and c;. = —c23. Substituting, we get 

tooCo3 + tes = — bzCo y (8) 
from which 


ee log 
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and in the same way we find 
= —(tis + bi2l2s) 








c ey ? 

5 thi + tes as 
a tysCo3 — bio — toalis 

eS a ” 


Proceeding in this way we can deal with matrices of any size, calculating 
the entries above the diagonal and simply changing the signs for the corre- 
sponding entries below the diagonal. We append the results obtained by the 
precise and approximate methods for a simple example. It will be seen that 
there is an error of the order of .02, which is quite as good an agreement as we 
are likely to get from two simple-structure rotations. 


TABLE 1 
Approximate Solution 
1.00 —.12 —.11 
1.01 oe Ta 
1.23 
By formulas: 
1.000 .066 .066 
— .066 1.000 .317 Csi 
— .066 —.317 1.000 
Normalizing: 
.991 .063 .063 
— .065 .951 .302 C,;D 
— .065 . 302 .951 
Symmetric transformation matrix: 
1.006 — .018 — .078 
— .019 1.175 — .370 TsjCi,D = KF ' 
— .080 — .371 1.170 
TABLE 2 


Precise Solution 


Original interfactor Correlation: 


1.00 12 16 
12 1.00 .58 is 
16 .58 1.00 


Symmetric transformation matrix: 
1.008 — .039 — .062 
— .039 1.174 — .373 pte 
— .062 — .373 1.176 
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METHODS OF SOLVING 
SOME PERSONNEL-CLASSIFICATION PROBLEMS* 


D. F. Voraw, Jr. 


YALE UNIVERSITY 


The personnel-classification problems considered in this paper are related 
to those studied by Brogden (2), Lord (6), and Thorndike (8). Section 1 gives 
an approach to personnel classification. A basic problem and variations of it 
are treated in section 2; and the computation of a solution is illustrated in 
section 3. Two extensions of the basic problem are presented in section 4. 
Most of the methods indicated for computing solutions are applications of the 
“simplex”? method used in linear programming (see 1, Chs. XXII, XXIII). 
The capabilities of a high speed computer in regard to the simplex method are 
discussed briefly (see section 1). 


1. Introduction. The type of problem to be considered is indicated by the 
illustration below. Consider two persons and two jobs, and suppose that 
the productivity of each person with regard to each job is known. Suppose 
further that each job must be filled and that each person must be assigned to 
one and only one job. It is assumed that the productivities can be repre- 
sented as single numbers. For example, let them be as follows: 























Jobs 
1 | 2 
(1:1) 
a/1/9|7 
j=} 
z 
42) 81s 





Thus 9 is the productivity of person 1 on job 1, 7 is the productivity of person 1 
on job 2, etc. Tomake maximum use of manpower the assigning agency would 
wish to establish an order of preference for the possible assignments and 
choose the assignment that heads the order. It would be natural to base 
such an ordering on average productivity. In the case at hand there are 

*This work was supported in part by the United States Air Force under Contract 
AF 33(038)-25192, monitored by the Personnel Research Laboratory of the Air Training 
Command Human Resources Research Center. Permission is granted for reproduction, 
translation, publication, use, and disposal in whole or in part by or for the United States 
Government. 
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two possible assignments. From (1:1) we obtain the following averages of 
the productivities associated with them: 


Assignment Average Productivity 
Person 1 on Job 1, Person 2 on Job 2 (9+ 5)/2=7 
Person 1 on Job 2, Person 2 on Job 1 (7 + 8)/2 = 7.5 (1:2) 


It follows that the preferred assignment is formed by placing person 1 on 
job 2 and person 2 on job 1. This gives the maximum possible average, 
which is 7.5. It should be noted that comparing assignments on the basis of 
the average productivity is equivalent to comparing them on the basis of the 
sum of productivities (see (1:2)). 

The situation described above involving two persons and two jobs sug- 
gests the following general problem: Given N persons, N jobs, and the pro- 
ductivity of each person on each job; find an assignment of persons to jobs 
such that the average productivity isa maximum. (Recall that it is assumed 
that each job must be filled and that each person must be assigned to one and 
only one job.)* Problems similar to this have been considered by Brogden (2), 
Lord (6), and Thorndike (8, p. 217). A basic form of the problem and certain 
modifications of it are discussed in section 2, Section 3 gives an illustration 
of a step-by-step computation of a solution. Extensions of the basic problem 
are formulated in section 4. 

Occasionally in personnel classification (e.g., in the military) there may 
be fewer job categories than jobs and/or fewer personnel categories than 
persons. This possibility has been taken into account in section 2 in the 
formulation of the basic problem. The N persons are considered to be such 
that a, are alike, a, are alike, --- , a, are alike (>>7., a; = N); thus there 
are only m distinct personnel categories. Similarly, regarding the N jobs 
it is assumed that b, are alike, b. are alike, --- , b, are alike + b; = N); 
thus there are only n distinct job categories. (m = n = N = 2 anda, = 
a, = b, = b, = 1 in the illustration associated with (1:1)). If, say, x;; persons 
of type z are placed on jobs of type J, their contribution to the “group pro- 
ductivity” is assumed to be c,;2;; , where c,; denotes the productivity (or 
expected productivity) of the 7th type of person on the jth type of job (7 = 1, 
-++,m;j = 1,--+,n). Fora discussion of how the c;;’s might be determined 
see (2, 8); in some situations c;; might be considered as a monetary saving. 

Two extensions of the basic problem will be considered (see section 4). 
In one of them the number, m, of personnel categories is assumed to be in- 
definitely large, but the number, n, of job categories is assumed to be finite. 
Each personnel category is represented as a point in n-space, where the n 
coordinates of the point represent the n productivities associated with the 
category. An n-dimensional distribution is assumed to be associated with the 


*In an actual situation the numbers of persons and jobs might be unequal; however, 
equality could be forced by introducing ‘‘dummy” jobs or persons. 
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set of personnel categories. The essential characteristic of the second exten- 
sion is that a vector c,,;;, °** , Ck,:; replaces the quantity ¢c,;. This extension 
might be of interest when, say, skill and preference of each person regarding 
each job must be considered separately; e.g., c,,;; could represent skill and 
¢o,;; could represent preference (here k = 2). 

Most of the computational techniques suggested in this paper originated 
in linear programming. Hand computation of the solution of a general linear 
programming problem is illustrated in 1, Chapter XXII. It seems possible 
that an “analogy” machine for solving such problems could be developed 
(see 3, p. 73). The example in section 3 illustrates hand computation of a 
solution of the special linear programming problem referred to in section 2 
(see also 1, Chapter XXIII). The National Bureau of Standards Eastern 
Automatic Computer (known as the SEAC) is coded for carrying out the 
simplex method used in the illustration. The SEAC is capable of solving a 
personnel-classification problem within a few hours when m + n < 60. It 
has obtained solutions of problems in which m = 4, n = 9, and m = n = 10 
in six minutes and twenty minutes, respectively. Government and military 
agencies have access to the SEAC through the National Bureau of Standards. 


2. A Basic Problem in Personnel Classification. Suppose that with regard to 
N persons there are m mutually exclusive personnel categories, and that with 
regard to N jobs there are » mutually exclusive job categories. Let a; and b; 
be, respectively, the number of persons in the 7th category and the number of 
jobs in the jth category (¢ = 1, ---,mjj7 = 1, ---,m). Assume that any two 
persons in the same category are essentially identical and that any two jobs 
in the same category are essentially identical. (This assumption involves no 
loss of generality since the case in which a job category or personnel category 
contains only one member is not excluded.) Let c;; be the productivity 
(or expected productivity) of any person in the ith category on any job in the 
jth category. Let 2,; represent any number of persons in the 7th category who 
can be placed on jobs in the jth category; the array (z;,) is termed an “‘alloca- 
tion.” Let T = >°,., ¢:;4:; , which might be considered as the group pro- 
ductivity (or expected group productivity) associated with (x;;). It is re- 
quired to find an allocation for which 7 is maximized (see 8, p. 217). More 
specifically, the problem* is to find values x)), +--+ , i, Of 211, °** , mn, 
respectively, for which 7’ assumes its maximum value, subject to the con- 
ditions: 


Do tis = 0; , 
t 
De tis = 4; , 
7 


*In regard to this problem the author wishes to acknowledge valuable discussions with 
S. Kakutani, J. W. Tukey, M. A. Woodbury, and J. T. Dailey. 


(¢ = 1, +--+, mez = 1, --+,n), (2:1) 
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[in (2:1) each a; , b; , and z,; is a non-negative integer and )>; a; = >; = 
N]. This can be shown to be equivalent (except in minor respects) to the 
Hitchcock-Koopmans transportation problem, a special topic in linear pro- 
gramming, which can be solved by means of the simplex method (see 1, 
Chapter XXIII). Professor J. von Neumann has shown* that this problem 
is essentially the same as that of finding a best strategy in a certain zero-sum 
two-person game having a payoff matrix of order (m + n) X mn. (See 7.) 

An interesting special case of the problem is that in which each c;; has 
only two possible values,t say 1 and 0 (which, for example, could mean 
“qualified” and “not qualified,” respectively). A variant of this special case 
is to determine whether N is the maximum of 7—and, if so, to find an alloca- 
tion for which T = N. (If the maximum of T is less than N, presumably the 
assigning agency would change one or more of the numbers a, , --: , a, , 
b, , --: , 6, or lower the standards regarding qualifications of persons for 
jobs.) The special case and this variant can be solved by means of Theorem 
3 in (5). 

A problem entirely similar to the one stated above (2:1) arises when 
c,; is considered not as a p:oductivity but as the cost of training any person 
in the 7th category to do any job in the jth category. In this situation T 
represents the total cost of training, and one wishes to find an allocation for 
which T assumes its minimum value. 

When N is large, one might wish to consider the numbers a, , «++ , d, 
by, ++, On, 2115 °** » Umn aS population proportions; the conditions stated in 
(2:1) would then be replaced by the following conditions: 


‘es t= b; ’ 
ti =O, (2:2) 
Ui; > 0, 


+o = : b = 1;a;,6; >0;2 = 1, ---, mj =1,---,n). Theuse 
of (2:2) in place of (2:1) introduces only minor modifications in the problems 
stated above. 


3. An Illustrative Example. In this section a step-by-step procedure for 
solving problems of the type presented in section 2 will be illustrated. The 
procedure is an application of the simplex method (1, Chapter XXIII). Each 
step of the method is characterized by a distinct allocation. Before setting 
up the illustrative example we shall consider a special property of these 
allocations and indicate the underlying idea of the method. 

A problem is said to be degenerate when a partial sum of a’s equals a 


*In a lecture at the Princeton University Game Seminar, October, 1951. 
tIr this case m < 2”. 











“er 
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partial sum of b’s (the equality >- 7a; = 9-10; does not satisfy this condition). = i 
(A degenerate problem can be replaced by an essentially equivalent one that “> — 
is non-degenerate.) In a non-degenerate problem each allocation used in the 
procedure has exactly m + n — 1 positive z,;’s. We shall denote them by 
Visisy 1 ' Cizie 7 = m+n — 1). With this set of r positive x; ;’s we associate 
quantities uw, , -** , Um, U1, °** Un, defined as follows: 


Uig + Vig = Cigig (g=1,-:-,7). (3:1) 


These equations can be solved for the w’s and v’s, where u, , say, is set equal 
to0. Let u; + v; = é@; , say. The allocation is optimal (i.e., >>;,, ¢:;2;; 
assumes its maximum value) if and only if 


Cis S Cay @Q=1,-++,m;7 =1,---,n). (3 :2) 


When the allocation associated with a given step is not optimal, one of its 
positive 2;;’s can be “discarded” and a new positive x;; can be “introduced” 
to form an allocation for the next step. The new z,; can be chosen as one for 
which the corresponding c;; yields a largest value of c;; — é;; . This choice 
determines which one of the positive x;;’s must be discarded. (See the discus- 
sion that follows the computing form below.) Every increase in ));,; ¢;;2%; 
from one step to the next is not less than a certain positive number, which 
depends on the ¢;,’s, a’s, and b’s. Since the maximum sum is finite, the pro- 
cedure involves at most a finite number of steps. 

The simplex method will be used to solve the problem stated below. 
Consider a large group of persons and a correspondingly large group of jobs. 
Assume that there are exactly four types of persons and exactly three types 
of jobs; thus m = 4and n = 3. Suppose that the productivities are as follows: 


[9 2 9 
18 8 
(¢;;) = (3:3) 
721 
9 8 0 








\ 


(c;; is the actual or expected productivity of a person of type 7 on a job of 
type j.) Let the proportions of the group associated with the four personnel 
categories be a, = .40, a, = .20, aj = .20, a, = .20; and let the proportions 
of the group required in the three job categories be b; = .35, b. = .35, b, = .30. 
(Note that }>t a; = >°} b;.) The problem is to find an allocation, (z;,), 
for which the group productivity, >>;,; ¢;;v;; , assumes its maximum value, 
where (2;;) is subject to the conditions given in (2:2). The allocation given in 
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(3:7) is a solution.* The problem is non-degenerate since no partial sum of a’s 
equals a partial sum of b’s (i.e., it is not the case that an a, a sum of two a’s, 
or a sum of three a’s equals a b or a sum of two b’s). 

All allocations used in the procedure have the property that exactly six 
of the z;,’s are positive. We may choose any such allocation as one with 
which to begin the procedure; accordingly, let the first one be as follows: 








; a’s 
35 0 .05 .40 
0 .15 .05 .20 
(2, i) ie (3 ‘4) 
0 0 .20 20 
0 .20 0) .20 


b’s 35 35 30 (1.00 » Sa, > »,) 


The six positive x,;’s in (3:4) ean be considered as having been chosen in the 
following sequence: 2; , 13 , L42 » Loo , Loz , X33 . The way in which each was 
chosen can be described as follows: choose any 2;; , say 2; , and set 2,, equal 
to its maximum possible value, .35, which is the minimum of a, = .40 and 
b, = .35 (this reduces the 4 X 3 array, 2;;, toa 4 X 2 array); next, in the 
reduced array choose any 2;; , say 2,3 , and set 2,3; equal to its maximum 
possible value, .05, which is the minimum of a, — 2,, = .05 and b, = .30; 
continue this method of selection until the original array has been completely 
reduced. 


We now introduce quantities u, , +++ , Us, 0; , ++ , ¥3 Which are to satisfy 
the following conditions [see (3:1)]: 
Wty =e, 


U + Us = C13, 

Us + 2 = Ca, 
(3:5) 

Uz + Ve = Coo , 

U2 + Us = Cos, 

Uz + U3 = C33 - 
The six c;;’s in (3:5) correspond to the six positive 2z,,;’s in (3:4). Solving 

(3:5) for U2 , Us , Us, V1, V2 , Vs We find that 

*Ifa,,°**,@4,0:, °°: , bshad been replaced by 100a , -*: , 100a, , 100b, , --* , 100b;, 


we could substitute the conditions in (2:1) for those in (2:2). A solution of this new problem 
would then be given by multiplying each 2;; in (3:7) by 100. 
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Oy = Gy er 5 
Vo = Cop — Cog tCizs — Uh , 
Ug = Cig. 4G (3:6) 
Uz = C3 — C3 +, 
Us = C33 — Cis +1, 
Us = Cyn — Cop + C23 — C2 + UY. 


For example, from (3:5) we have that v3 = c,3 — WU, , Us = C33 — V3 = C33 — 
Ci3 + u, , etc. The allocation in (3:4) is optimal if and only if for every 
(i, J) €i; = cs; , Where ij = u,; + 0; [see (3:2)]. It will be noted from Step 1 
on the computing form below that this allocation is not optimal: e.g., é,, = 
1 < cs, = 7. This requires that a second step of the procedure be performed. 
The allocation associated with the second step will be formed from (3:4) 
by discarding x;,; and introducing 23, [see the remarks immediately below 
(3:2)]. 

The information in (3:6) and the values of the ¢,;’s are given in Step 1 
of the computing form below. The first column of the form gives the positive 
z,;'8; the next column gives the values of the corresponding c,;’s. In the 
columns headed “Coefficients of u,, +--+ , v;’’ the +’s and —’s given in (3:6) 
are indicated, where u, has been equated to 0. Using the c;; column and the 
+’s and —’s in a u or v column, we can evaluate the wu or v rapidly; e.g., 
w= -9-—8+8+8 = —1 (see the last row of the form for Step 1). The 
last column gives the products ¢;;2,; and their sum. The way of passing from 
one step to the next is indicated below in discussions of the steps. An interest- 
ing feature of the general procedure is that for any step each coefficient in the 
expression of a u or v as a linear combination of c’s is +1, 0, or —1. The 
computing form is a slight modification of one given in (4). 

Discussion of Step 1. It is not the case in Step 1 that each é;; > c,; ; 
thus another step of the procedure must be performed. The maximum of 
the differences c;; — €;; is C3, — &3, = 7 — 1 = 6, and so 2;, will be given a 
positive value in the allocation for Step 2 (note that é;, has been italicized in 
Step 1). Since é, = us + v, , we have that é, = c,, + ¢3; — C3 (thus 
31 + Cis = Cy + C33). This indicates that by adding, say, d > 0 to x3, and 
3 in (3:4) and subtracting d from 2,, and 23; a new allocation will be formed 
which differs from (3:4) in that the new 2;, will be positive and, if d is suffi- 
ciently large, the new 2, or 233 will equal 0. Since in Step 1 233; = .20 is 
smaller than z,, = .35, we set d equal to .20 (thus the new z,; will be 0). It 
follows from the expression above for é3, that c33 = €3: + ¢i3 — Ci: ; from this 
we can easily determine the coefficients of the w’s and v’s for Step 2; e.g., in 
Step 1 us = c33 — C3, thus in Step 2 us = C33 + C13 — C11 — C13 = C31 — Cn - 

Discussion of Step 2. The é;;’s obtained in Step 2 are not such that each 
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€;; > ¢;; ; hence a third step of the procedure must be performed. There igs 
one (and only one) case in which c;; — €,; is positive—namely,.c., — é, = 
9 — 8 = 1; thus 2,, should be positive in the new allocation. (Note that 
€,, is italicized in Step 2.) Since G4 + ¢i3 + Coo = C1, + Cos + Ca , it follows 
that .05 can be added to 24, , Z;3 , and 22. and subtracted from 2, , 23 , and 
X42 (X23’8 value becomes 0). It should be noted that c.,; = 4 + Cis + C2 — 
Ci1 — C42 ; by means of this we can easily determine coefficients of the u’s 
and v’s for Step 3. 

Discussion of Step 3. The c,,;’s associated with Step 3 are such that each 
€;; > c;; ; thus the procedure ends with this step, and the allocation involved 
is a solution of the problem. The sum (8.25) obtained is at least as large as 
the sum associated with any other allocation. From the first column of Step 3 
we have that the solution, (x;;), is as follows: 


‘10 60. .30) 
0 2 0 
(z;;) = : (3:7) 
20 0 «0 
05 15 0 





An optimal classification of the group of persons is formed when the persons 
of type z assigned to jobs of type j constitute a proportion 2,; of the total 
group, where z,; is given in (3:7) (¢ = 1, --- ,4;7 = 1, ---,3). The average 
productivity associated with this optimal classification is 8.25. 


4. Extensions of the Problem. Two different extensions will be presented. 
The first involves a population of personnel categories with which an n- 
dimensional distribution is associated; the second involves replacement of the 
scalar, c;; , by a vector. 


Let S be an n-space, and represent any point in S by (z, , -+- , @). 
Represent the ith row of (c;;) as a point, (¢;,, «++ , ¢,,), in S (¢ = 1, «++ , m) 
and let F,,(z; , -*: , 2,) be the distribution function associated with the m 


points representing the m rows of (c,;); thus, in the problem associated 
with (2:2) 


Fn(@: 5 °°* 5%) = DL, (4:1) 

where q ranges over all values of 7 such that for every j, ¢;; < 2; . Fula, 
- , z,) is the probability that the n productivities, say c, , «++ , C, , of an 
individual selected at random would be such that c; < z; (j = 1, «+: , ”). 
F,,(2: , *** , 2n) is a discontinuous distribution; an extension of the problem 
arises when the population is such that the distribution, say F(z, , «++ , 2n)s 


is continuous (e.g., a cumulative normal n-variate distribution). This exten- 
sion is considered in (2), (6), and (8). The problem is to find n mutually ex- 
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clusive regions, say R, , --- , R, , in S such that } Se, 2, dF is maximized 
subject to the restrictions: 


[ ar=b,, (6 20; DG =1;7=1,--,m). (42) 
“Rj 7 


The optimal classification of the population would consist in assigning all 
individuals in R; to jobs in the jth category (7 = 1, --- , n). It follows from 
(6) that the regions would be determined by solving n simultaneous equations 
in n unknowns. A solution can be obtained very easily when n = 2 (e.g., 
see 8, pp. 218-219). An approximate solution would be obtained by 
“srouping’’—i.e., by approximating F(z, , --- , z,) by a discontinuous distribu- 
tion, say F(z, , «++ , 2,)—and then solving a problem of the type associated 
with (2:2). The quantity a; in (2:2) would represent the amount of a “‘jump”’ 


in the approximating distribution, F,,(z, , «++ , 2.) (@ = 1, +--+ , m). 
In the first problem stated in section 2, let N be large and let c;; be re- 
placed by a vector (¢1,;; , *** , Css) @ = 1, ---,mj = 1,---,n). (See 


the remarks in the next-to-last paragraph of section 1 regarding a case where 
kwould equal 2.) Let Ty = os,; ¢n.:;%i; (h = 1, +++, k)jandletR = DirgaT 
be a composite indicator of the value of an allocation (x;;) (9: , --: , ge are 
weights). Note that R = >o;,; w.;2:, , where w,; = Dox gata: 3 Ws; is like 
the c;; in section 2. The problem is as follows: Find an allocation for which 
R assumes its maximum value, where 2; , --* , mn, 1, °°*»Qm,01,°°* , Op 
are subject to (2:2) and to inequalities of the form 7’, > e, (here e, represent 
a minimum acceptable value of 7, ; however, the problem is essentially 
unchanged if for any h the inequality is reversed). A preliminary is to de- 
termine whether there is any allocation satisfying the specified conditions. 

The problem stated above is an example of a linear programming problem, 
which can be solved by means of the simplex method (see 1, Chapters XX, 
XXIJ, XXII). 
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Addition at proof reading: The simplex method provides a general solution of the basic 
problem stated in section 2; however, it may be awkward when m or n is large (see section 
3). P. S. Dwyer has proposed some methods including a modification of the simplex 
method. T. E. Easterfield’s paper (A combinatorial algorithm, J. Lond. Math. Soc., 1946, 
21, 219-226) gives another interesting method. Procedures for obtaining approximate 
solutions are given in section III of a forthcoming bulletin by D. F. Votaw, Jr., and J. T. 
Dailey (Assignment of Personnel to Jobs, Human Resources Research Center, Lackland 
Air Force Base, San Antonio, Texas). 
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THE CLOSURE FACTORS RELATED TO 
OTHER COGNITIVE PROCESSES* 


CaRroL PEMBERTON 


UNIVERSITY OF DELAWARE 


Twenty-five group tests, assembled with certain hypotheses concerning 
the first and second closure factors in mind, were administered to 154 subjects, 
mostly graduate students. The intercorrelations were analyzed factorially, 
yielding eight factors that were rotated to an oblique simple structure. The 
factors were interpreted as: speed of closure, C; ; flexibility of closure, Ce ; 
verbal closure, C; ; word fluency, W; reasoning, R; perceptual speed, P; the 
first space factor, S; ; and speed of handwriting, H. Four second-order factors 
were tentatively described as analytical ability, synthetic ability, speed of 
perception, and word fluency. Three of the reasoning tests had their highest 
loadings on Cz and one on C;, which seems to be evidence that flexibility of 
= generalizes in the cognitive domain and is associated with analytical 
ability. 


I. Purpose of the Study 


Factors named by Thurstone “speed of closure” and “flexibility of 
closure’ have been isolated in several previous studies (1, 2, 17, 19). The 
purpose of the present study was to ascertain whether abilities on speed of 
closure and flexibility of closure tests generalize to other domains. The part 
of the study to be reported in this paper deals with generalization of the 
closure factors to tasks requiring higher cognitive functions. The generaliza- 
tion of the closure factors to the conative and affective domains will be dis- 
cussed in a subsequent paper. 

The hope of finding meaningful personality variables through studying 
perception has been expressed by Klein and Schlesinger (9), and by Frenkel- 
Brunswik (6), among others. 

Thurstone (19) writes: 


_ _ The first closure factor C, (speed of closure) seems to facilitate the making of a closure 
in an unorganized field, the second closure factor C, (flexibility of closure) seems to facilitate 
the retention of a figure in a distracting field. If this .. . interpretation of the two closure 
factors has any generality beyond the perceptual domain, then one could imagine that the 
factor C, determines the ease with which the subject can unify a complex situation, whereas 
the second factor determines the ease with which he can keep in mind its essential features 
against distraction... The first closure factor might be associated with inductive thinking, 
whereas the second closure factor might be more associated with deductive thinking. 


*This paper summarizes part of a dissertation submitted in partial fulfillment of the 
requirements for the degree of Doctor of Philosophy at the University of Chicago. The 
writer is deeply indebted to Dr. L. L. Thurstone for his generous advice and guidance. 
The complete study is obtainable on microfilm, from the University of Chicago library, 
Film No. T1279 (price $2.15). It includes reproductions of the tests used, score distribu- 
tions, and plots of the oblique V-matrices. 
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The present investigation deals with a factor analysis of the intercorrela- 
tions of twenty-five tests selected to include longer forms of tests for both 
closure factors; inductive and deductive reasoning tests; verbal closure tests; 
and key tests for the first space factor, perceptual speed, word fluency, and 
speed of handwriting. 


II. Description of the Tests 


1. Figures. Twenty rows of geometrical figures are presented, with in- 
structions to mark the figures of the six on the right which can be made the 
same as the one on the left by sliding them around in the plane of the paper. 

2. Cards. A hundred and twenty diagrams of differently shaped cards 
are given and the subject asked which cards of the six in a row can be made 
the same as the model by sliding them around in the plane of the paper. 

3. Gestalt Completion. This test is similar to Thurstone’s adaptation of 
the Street Gestalt Completion test (12). Two 24-item forms of this test were 
devised by the writer and the scores for the two combined. 

4. Mutilated Words. Two 51-item forms similar to Thurstone’s test (17) 
were prepared by the writer and the scores for the two combined. 

5. Concealed Figures. This 49-row form of the Gottschaldt Figures Test 
was devised in the Psychometric Laboratory, University of Chicago. A small 
design at the beginning of each row is followed by four more complex designs. 
The subject indicates in which of the more complex diagrams the simple figure 
occurs. 

6. Designs. In this test of Thurstone’s (15), 300 designs are presented, 
in 40 of which the Greek capital letter ‘‘sigma” is concealed. The task is to 
mark the figures containing the ‘“‘sigma.’’ 

7. First Letter. The subject lists all the words he can that begin with 
the letter ‘“‘c.” 

8. Suffixes. The subject lists all the words he can that end with the 
letters “‘-tion.” 

9. Writing Phrase. The task here is to write the phrase ‘‘Now is the time 
for all good men” as many times as possible within one minute. 

10. X’s. The subject writes as many X’s as possible in 30 seconds. 

11. Identical Forms. The first figure in each of 60 rows of figures is 
exactly the same as one of the five numbered figures following it. For each 
row the subject writes the number of the figure which is identical with the 
model. 

12. Identical Numbers. The subject is required to mark the number 927 
each time it occurs in 18 columns of three-digit numbers. 

13. Number Series. The subject is to discover the rule underlying each of 
22 series of numbers and fill in the blanks. 

14. Letter Series. This 25-series test is similar to the previous one, except 
that letters are used. 
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15. Figure Classificaiion. This is a test of Spearman’s adapted by Thur- 
stone (14). Two groups of four figures each occur at the beginning of each 
line, followed by eight more figures. The subject is to discover the rule by 
which the first two groups were constructed, and then mark those of the re- 
maining eight figures which belong to Group I. 

16. False Premises. The subject labels as true or false each of 25 syllo- 
gisms composed of absurd-sounding premises and conclusions. 

17. Hidden Words. This test was devised for the present study. A 
four-letter word appears at the beginning of each of 36 sentences. The subject 
is to encircle the four-letter word whenever it occurs, either embedded in a 
longer word, or in two longer words, ignoring spaces and punctuation. This 
test represents an attempt to produce a verbal test of flexibility of closure. 
The sentences were made amusing in an effort to strengthen the Gestalt of the 
complete sentence. 

18. Incomplete Words. Individual letters are deleted from 50 common 
English words and the subject is asked to fill in the missing letters. 

19. Anagrams. Fifty English words are presented, the letters of which, 
when rearranged, spell another English word. The subject encircles the letter 
with which the new word begins. The words to be found are well-known ones, 
the majority being AA or A in Thorndike’s (13) word list. 

20. Scrambled Words. This is a revision of a test constructed by Botzum 
(2). The letters of 50 common four-letter words are rearranged so as to form 
nonsense syllables. The subject encircles the first letter of the unscrambled 
meaningful word. 

21. Copying. This test was adapted by Thurstone from a test by Mac- 
Quarrie (14). The subject is to copy 36 figures in the dotted spaces provided. 
Each drawing must have the same size, shape, and position as the original. 

22. Four Letter Words. This test, devised by Bechtoldt (1), and revised 
by Botzum (2), consists of rows of evenly-spaced typewritten capital letters. 
In each row there are a few combinations of four letters that spell words, which 
are to be encircled. 

23. Backward Writing. In this test, designed by Thurstone (16), 50 words 
are typed in the normal] manner, and below each are four words reversed as in a 
mirror. The subject underlines the word corresponding to the normally typed 
one. 

24. Hidden Pictures. This test of Thurstone’s contains items like those 
found in children’s books, in which one ean see objects hidden in the lines of a 
larger picture. Six pictures, in which 28 hidden persons are to be located, 
constitute the test. 

25. Sentences. A test similar to this was used by Meili (10). In the present 
form, devised for this study, the subject is required to write as many sen- 
tences as he can containing each of four given words, then as many as he can 
containing each of five and six given words. 
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Data on reliability coefficients, means, standard deviations, time limits, 
and scoring formulas are summarized in Table 1. The reliabilities are cor- 
rected split-half coefficients. As the tests are all speed tests, the reliability 
coefficients based on parallel forms would probably be lower. Distributions 
were all approximately normal except for the test Backward Writing, where 
there was a bunching of high scores as a result of a too liberal time limit. 


III. Subjects and Administration of the Tests 


The 154 subjects (122 graduate students and 32 undergraduate students) 
were volunteers, obtained through announcements on bulletin boards at the 
University of Chicago. There were 94 men and 60 women. All the divisions 
and professional schools were represented, but the largest group (43 persons) 
was from the Social Science Division. Ages ranged from 17 to 59 years, 
with the majority of cases between 19 and 29, and the average 26.4 years. 


TABLE 1 
Information Regarding the Tests 








Time Limits 
(minutes) Scoring 
Test Mean Ss. D. r Formula 
Fore-Ex. Test 








1. Figures 20.97 9.40 .96 2 3.5 R-W 
2. Cards 27.46 12.09 .96 2 4 R-W 
3. Gestalt Completion 35.98 6.94 .80 74 3 R 

4. Mutilated Words 72.01 14.14 .87 2 5 R 

5. Concealed Figures 100.88 28.70 .94 4 10 R-W 
6. Designs 26.94 6.28 94 2 3.5 R 

7. First Letter 56.98 13.36 _ 2 5.5 Total 

8. Suffixes 28.44 9.00 _— 1.5 5 Total 

9. Writing Phrase 54.05 6.97 — 1 1 Total 

10. X’s 62.68 8.22 _ 5 5 Total 

11. Identical Forms 44.68 9.11 -98 3 4 R — W/4 
12. Identical Numbers 63.74 7.02 .92 3 2.5 R 

13. Number Series 11.66 3.53 .82 4 7 R 

14. Letter Series 13.55 4.28 .85 4 6 R 

15. Figure Classification 66.79 23.22 .94 6 10.5 R-W 
16. False Premises 12.33 5.54 .65 3 6 R-W 
17. Hidden Words 68.94 15.64 .98 3 5 R 

18. Incomplete Words 29.31 9.80 .95 1.5 5 R 

19. Anagrams 18.74 7.59 .88 4 6.5 R 
20. Scrambled Words 27.21 10.65 .94 4 5 R -— W/3 
21. Copying 16.36 7.93 -96 4 4 R 
22. Four-Letter Words 39.32 9.11 .92 3 4.5 R 

23. Backward Writing 40.74 7.98 .98 2 3 R —- W/3 
24. Hidden Pictures 13.31 3.35 .66 5 14 R 
25. Sentences 12.60 3.71 _ 1 9 Total 
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Tests were administered in group form on three consecutive evenings 
during August, 1950, from 7:30 to 10 p.m. 


IV. The Factor Analysis 


All tests were scored by hand twice, usually by two different people. 
Pearson product-moment correlation coefficients were calculated from the raw 
scores by I.B.M. equipment (Table 2). Eight factors were extracted from the 
correlation matrix by the complete centroid method (18). The matrix was 
factored twice. During the first factoring, the highest value in each column, 
adjusted after the extraction of each factor, was used as the diagonal entry. 
During the second factoring, the communalities resulting from the first factor- 
ing wereemployed. The largest difference between the communality estimates 
resulting from the two factorings is .06, and the majority of differences are 
.00 or .01. The eighth-factor residuals (Table 3) range from .08 to —.10 
with a mean of .00 and a standard deviation of .030. The orthogonal centroid 
matrix F is reproduced in Table 4. 

The F-matrix was rotated to a simple structure using both radial and 
single-plane methods of rotation (18). Table 5 presents the final transforma- 
tion matrix and Table 6 the resulting oblique factor matrix, V. Table 7 
shows the cosines between the reference vectors of the V-matrix, and Table 8 
the correlations between the primary axes. The reference vectors have been 
identified by the letters A to H, but in Tables 8-12, which refer to the primary 
vectors, the columns are denoted by letters which refer to the psychological 
interpretation of the factors. The columns are kept in the same order to 
facilitate the comparison of tables. 

Table 8 shows that some of the correlations between the primary vectors 
are high. For this reason the A-matrix presented in Table 11 was calculated. 
This table represents the test vectors as linear combinations of the primary 
vectors. It is obtained by post-multiplying the V-matrix by the inverse of 
the diagonal matrix, D. The values of D (Table 9) are also used in computing 
the matrix T (Table 10), which represents the eight direction cosines of each 
primary vector. Table 12 gives the correlations between each primary factor 
and each of the tests. 


V. The Second-Order Analysis 


The complete centroid method of factoring was used, and Table 8 was 
factored three times to stabilize the communalities. Table 13 shows the 
orthogonal matrix F, ; the residuals after five factors had been extracted 
appear in Table 14. The centroid matrix was rotated to an oblique simple 
structure. The oblique matrix, V, , and the transformation matrix, A, , are 
shown in Tables 15 and 16, and the cosines between the reference vectors in 
Table 17. Because of the impossibility of determining the second-order 
primary vectors with much confidence, the correlations between the primaries 
were not calculated. 
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VI. Interpretation of the First-Order Factors 


Primary factors are usually interpreted by referring to loadings on the 
reference vectors. However, the aim of factor analysis is to express the test 
vectors as linear combinations of the primary vectors, and this is given by the 
A-matrix (Table 11) rather than the V-matrix. In an oblique solution the 
A-matrix is proportional to the V-matrix by columns but not by rows. A 
test may have its highest loading on a given reference vector, but its highest 
component is not necessarily of the corresponding primary vector. 

For example, Test 4 had its highest loading on reference vector F, and its 
second highest on A, whereas in the A-matrix its largest component is of the 
primary vector C; , which corresponds to the reference vector A. A more 
marked reversal occurs in the case of Test 14. Due to our inadequate knowl- 
edge of the significance of factor loadings, it is difficult to say whether these 
reversals are important. However, because it seems more logical to interpret 
from the A-matrix, this has been done. Since most people interpret from the 
V-matrix, the loadings on the reference vectors, as well as the components of 
the primary vectors, will be quoted. 


TABLE 3 
Distribution of Eighth-Factor Residuals* 











Residual Frequency 
.08 4 
.07 8 
.06 19 
.05 18 
.04 27 
.03 47 
.02 80 
.O1 90 
.00 104 

—.01 59 
— .02 52 
— .03 52 
— .04 26 
— .05 22 
— .06 9 
— .07 0 
— .08 4 
—.09 
—.10 4 
N = 625 





*This table includes the residuals from the diagonal entires. 
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VI. Interpretation of the First-Order Factors 

Primary factors are usually interpreted by referring to loadings on the 
reference vectors. However, the aim of factor analysis is to express the test 
vectors as linear combinations of the primary vectors, and this is given by the 
A-matrix (Table 11) rather than the V-matrix. In an oblique solution the 
A-matrix is proportional to the V-matrix by columns but not by rows. A 
test may have its highest loading on a given reference vector, but its highest 
component is not necessarily of the corresponding primary vector. 

For example, Test 4 had its highest loading on reference vector F, and its 
second highest on A, whereas in the A-matrix its largest component is of the 
primary vector C; , which corresponds to the reference vector A. A more 
marked reversal occurs in the case of Test 14. Due to our inadequate knowl- 
edge of the significance of factor loadings, it is difficult to say whether these 
reversals are important. However, because it seems more logical to interpret 
from the A-matrix, this has been done. Since most people interpret from the 
V-matrix, the loadings on the reference vectors, as well as the components of 
the primary vectors, will be quoted. 


TABLE 3 
Distribution of Eighth-Factor Residuals* 











Residual Frequency 
.08 4 
.07 8 
.06 19 
.05 18 
.O4 27 

..03 47 
.02 80 
01 90 
.00 104 

— .0] 59 
— .02 52 
— .03 52 
— .04 26 
—.05 22 
— .06 9 
—.07 0 
— .08 f 
— .09 0 
—.10 t 

N = 625 





*This table includes the residuals from the diagonal entires. 
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1. Verbal Closure, C3 


Test Loadings on A Components of C3 

19. Anagrams .54 91 
20. Scrambled Words 52 .88 

18. Incomplete Words .48 81 

4. Mutilated Words .38 .64 

22. Four-Letter Words 36 .61 

17. Hidden Words 34 .57 

14. Letter Series .33 . 56 

12. Identical Numbers 82 54 


The tests which are high on this factor all require speed in organizing 
or reorganizing a set of highly practiced symbols, such as letters or numbers. 
No spatial or pictorial tests are represented. The factor has a high correlation 
(.70) with the word fluency factor, W, and the four tests with the highest 
loadings all require the subject to think of words rapidly, and to select the one 
which fits the restrictions set by the problem. However, these restrictions are 
more rigid than those for the W tests. In the tasks high on C; only one word 


TABLE 4 
Orthogonal Centroid Factor Matrix, F 











Test I II III IV V VI VII Vill h? 
1 53 38 05 a0 .30 16 27 21 67 
2 66 43 13> 26 .28 06 16 15 77 
3 46 47 —.13 28 18 -—.17 —.09 05 60 
4 68 —.03 .04 0 =—.21 —.27 =—.15 11 63 
5 70 48 —.12 —.14 —.23 07 10 —.09 83 
6 65 10 —.22 07 —.06 09 07 —.20 54 
7 550 —.41 .06 —.20 18 —.08 23 «—.12 62 
8 40 —.42 21 —.10 16 —.25 10 —.21 .53 
9 17 —.87 -—.89 —.39 04 .05 —.19 .20 00 
10 23 —.18 —.20 —.27 —.07 18 31 18 35 
11 59 23 =—.3l .18 .30 06 —.04 —.11 64 
12 58 —.25 —.25 .26 —.04 os -—.10 —.11 50 
13 57 14 39 -—.02 —.0 06 —.17 —.06 54 
14 66 —.04 25 14 15 15 —.31 —.03 66 
15 51 32 34 =—.14 — 07 0 =—.16 =—08 54 
16 32 18 29 —.24 —.20 04 —.20 —.04 36 
17 7a 6 -—.26 0 —.24 .21 .04 9 =O01  =—-06 72 
18 67 —.28 26 04 —.22 -—.12 —.02 11 67 
19 65 —.38 23 18 —.14 07 05 07 68 
20 63 —.39 37 20 «—.18 11 20 =—.03 80 
21 62 a =. =19 —26 —.12 02 02 63 
22 67 -—.24 —.21 OF —.17 08 03 03 59 
23 68 —.12 —.02 .10 es, 12 —-.11 53 
24 27 23 = a3 i) C Mf 05 09 38 
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will fit the requirements, while in the W tests there are numerous acceptable 
words. 

Mutilated Words and Incomplete Words both have loadings on C, 
(identified as speed of closure), as well as on C;. In these two tests the ele- 
ments to be combined are given in their correct order, and the task can be 
performed by a rapid synthesis. However, while administering Mutilated 
Words as an individual test, the writer observed that many subjects did this 
task analytically, one subject remarking that he was “learning to break the 
code.” Moreover, if the word is not seen quickly and correctly as a whole, 
then the ability to change one’s set rapidly seems to be important. Due to in- 
ability to relinquish an incorrect response, many people perform inadequately. 

In Anagrams the subject must abandon the Gestalt which is formed by 
one English word and reorganize the letters into another word. Similarly, 
in Scrambled Words some of the nonsense syllables are pronounceable and 
form Gestalts that must be broken. In Four-Letter Words, Hidden Words, 
and Identical Numbers, one must find or retain a Gestalt in a distracting field. 
All these tests therefore seem to require elements of what has previously been 
called “‘flexibility of closure.”” The factor itself correlates .54 with flexibility 
of closure, C. . 

Letter Series has almost equal components of this factor and factor R. 
If R accounts for the synthetic aspects of reasoning, then one would expect 
Letter Series also to have a loading on a factor representing analytic ability. 
Similarly, Number Series, Figure Classification, and False Premises had load- 
ingson Rand C,. What differentiates C3; from C, seems to be the more highly 
practiced nature of the symbols involved in the C, tasks. 

There is a slight negative correlation between the C; factor and C,. 
This may be due to C, tests being more pictorial and spatial in nature, or it 
may be that the C; tests require flexibility of closure rather than speed of 
closure. Most of the tests do seem to require flexibility of closure; but in 


Botzum’s study, Four-Letter Words and Scrambled Words had loadings of 


TABLE 5 


Final Transformation Matrix, A 











A B C D E F G H 

I 266 .170 .215 .1138 . 166 123 .112 .097 
II —.389 359 .001 —.196 .302 065 104 —.227 
III Big .028 —.447 121 053 —.076 3386 —.302 
IV .529 —.505 .316 —.397 .044 167 —.159 —.543 
Vv —.3855 —.477 .061 . 430 .535 —.228 .198 . 182 
VI .145 .157 427 —.559 .109 —.809 219 . 103 
VII —.219 —.070 —.208 180 .413 —.093 —.834 —.396 


Vill .5623 —.574 —.652 —.497 .639 481 —.237 .594 
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TABLE 6 
Oblique Factor Matrix, V = FA 














Test A B C D K F G H 
1. Figures .03 —.08 .02 —.07 .64  —.02 —.08 —.02 
2. Cards — .06 .08 —.03 .07 .56 .03 .07 .04 
3. Gestalt Completion .02 —.04 17 —.04 .30 .27 .07 —.09 
4. Mutilated Words .38 .06 —.01 .02 —.02 43 .07 .07 
5. Concealed Figures — .07 .53 .21 —.05 .12 .05 .01 —.06 
6. Designs 05 .26 41 .03 01 —.05 -—.01 —.08 
7. First Letter .02 .00 .03 45 .07 —.06 —.07 .10 
8. Suffixes .03 —.03 —.02 .538 —.09 .04 .03 —.03 
9. Writing Phrase .06 —.03 .02 .06 —.04 .02 .04 64 
10. X’s .01 .06 —.06 —.03 .20 —.08 —.29 .25 
11. Identical Forms — .04 .01 44 .05 .24 —.02 .09 .01 
12. Identical Numbers 82 —.05 .41 —.02 —.12 .05 .O1 .02 
13. Number Series .19 Rs .03 .02 01 .00 386 —.06 
14. Letter Series .33 .02 .23 —.01 .07 —.06 -46 .07 
15. Figure Classification .03 .39 .03 .03 .05 —.04 .38 —.04 
16. False Premises .04 .39 —.07 .01 —.10 .00 .33 .06 
17. Hidden Words 34 —.03 .46 —.03 .03 —.08 01 .07 
18. Incomplete Words .48 .03 —.10 .06 —.03 25 .05 .05 
19. Anagrams .54 —.08 .06 —.02 01 .07 .01  —.01 
20. Scrambled Words 52 —.04 .05 -05 .02 —.04 —.04 —.18 
21. Copying .00 41 .04 01 .06 .24 —.01 .07 
22. Four-Letter Words .36 .06 .26 —.06 —.02 .08 —.08 .13 
23. Backward Writing 15 01 21 21 13 .05 —.02 —.05 
24. Hidden Pictures .02 .02 .06 —.05 43 89 —.22 —.03 
25. Sentences — .08 .O1 -06 25 me | .O1 .23 .42 





.34 and .26 on C, , whereas in this study their loadings on C, are negligible. 
Due to this ambiguity the factor has merely been named Verbal Closure, 
although a more accurate descriptive phrase might be ‘flexibility of closure, 
utilizing highly practiced symbols.” 


2. Flexibility of Closure, C2 


Test Loadings on B Components of C, 
5. Concealed Figures .53 ao 
21. Copying 41 61 
15. Figure Classification .39 .58 
16. False Premises .39 .58 
13. Number Series Zi .40 
6. Designs .26 .39 


This factor seems to correspond to Thurstone’s flexibility of closure, C2 
(17). Three of the tests are ones which defined this factor in the mechanical 
aptitude study (19), namely Copying, Designs, and Concealed Figures (a 
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TABLE 7 
Reference Vector Cosines, C = A’A 











A B C D E F G H 
A 1.000 -—.450 -—.108 -—.616 .028 318 —.017 .123 
B — .450 1.000 .291 182 —.518 <=—.3é61 .280 —.183 
y —.108 .291 NOOO 8 6=.082 =—.89% —.541 207 —.265 
D — .616 .132  —.082 1.000 —.125 .025 027. —.111 
q 023 —.018 =.d97 —.12b 1.000 103 —.304 . 232 
F 2018 —.331 —.541 .025 103 1.000 —.291 .127 
G = OF . 280 .257 02/ —.304 —.291 1.000 .221 
H 123 —.1838 —.265 —.111 . 232 .127 .221 1.000 





longer form of Gottschaldt Figures). In addition to these three tests we find 
three reasoning tests with significant loadings. 

The C, factor is interesting, since it cuts across the content of the test 
material. What seems to be required in all these tests is ‘freedom from 
Gestaltbindung,” the Gestalt being formed either by the objective stimulus or 
by the mental set adopted by the subject. In Concealed Figures one must 
break the Gestalt formed by the large figure in order to find the small figure. 
One must also ignore the optical illusions which are often created by the large 
figure, distorting the appearance of the small figure. 

It has been suggested that Copying has a loading on C, because the sub- 
ject must not be influenced by the regularly spaced dots in reproducing the 
given figure (19). This hypothesis could be investigated by giving the test 
without the dotted background, as well as with a more highly distracting 
background. The former method of presentation should decrease its loading 
on C, , the latter increase it. However, it is possible that this test would have 
a high loading on C,., whatever the method of presentation. The C, factor 
seems to represent the ability to abstract. According to Goldstein and 


TABLE 8 
Correlations between Primary Factors, R = D(A’A)~!D 











C; C2 P W Si CQ R H 
C3 1.000 .538 .078 . 704 .348 —.269 —.182 052 
C2 .538 1.000 Pei .333 .540 .092 —.175 .095 
P .078 mb eg 1.000 . 169 .393 .465 —.099 . 204 
W . 704 333 . 169 1.000 304 —.180 —.155 120 
Si 348 .540 353 3804 1.000 . 190 158 —.134 
C; — ,209 .092 .465 —.180 . 190 1.000 241 — .068 
R —.182 —.175 —.099 —.155 . 158 .241 1.000 —.376 


H .052 .095 . 204 120 —.134 —~.065  —.376 1.000 
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TABLE 9 
Diagonal Entries of Dip 








C3 C2 ig W Si C, R H 





d 594 .673 .740 .688 . 706 743 . 834 .877 





Scheerer (7) a characteristic of abstract behavior is “to grasp the essential of 
a given whole; to break up a given whole into parts, to isolate and to synthesize 
them.’”’ This seems a very good description of what one must do in Copying. 

In Figure Classification the subject must not be bound by his first 
hypothesis concerning the classification of the figures into two groups, nor 
perseverate with previous solutions inapplicable to subsequent items. In 
Number Series the subject must also test one hypothesis after another, apply- 
ing the four arithmetical operations of addition, subtraction, multiplication, 
and division. In False Premises the subject must not be influenced by the 
surface absurdity of the statements. “A monkey hanging from the branch of 
a tree is a fruit,” sounds a ridiculous assertion, but in terms of the given 
premises the conclusion is to be marked “True.” 

The Designs Test has a relatively low loading on this factor, its highest 
loading being on perceptual speed, P. Designs was devised as a perceptual 
speed test but in the first study in which it was used (15) it did not have a 
loading on P. Thurstone later hypothesized that Designs would have 
loading on C, , and this was borne out in the mechanical aptitude study (19). 
However, it now appears that for a more highly educated group of subjects 
the test is one which requires perceptual speed to a greater extent than 
flexibility of closure. 

The two reasoning tests which have their highest loadings on C, are the 
two tests which had the highest loadings on Botzum’s deduction factor. This 
tends to corroborate Thurstone’s hypothesis that C, is associated with deduc- 


TABLE 10 
Direction Cosines of Primary Vectors, 7 = DA7! 











I II III IV Vv VI VII VIII 
C; 741 —.480 .350 .078 —.219 .147 131 .074 
Cy 711 .279 112 —.386 —.452 .110 .187 —.060 
P .552 .092 —.678 .331 . 202 033 —.124 —.248 
W 647 —.504 229 —.194 .256 —.249 .219 —.249 
Si way .473 129 .037 .339 141 259 .207 
C; 243 494 —.354 227 —.098 —.6384 —.312 .109 
R —010 .380 .492 . 122 034 -—.026 —.695 -—.051 


H 144 —.386 —.581 —.579 .031 127 0.201 .316 
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tive reasoning (19). The writer prefers to differentiate the reasoning factors 
in terms of the methods of problem-solving involved, rather than in the 
epistemological terms of induction and deduction. Yela (20) maintains that 
the solution to reasoning problems can occur either “by closure of the un- 
finished configuration, perceiving the lacking element as connected with the 
others, or by analytically reasoning out the principle connecting the items.” 
This is not a new idea. Duncker (5) differentiates “analytic and synthetic 
evidence,” which, he contends, are not opposites. Synthetic evidence can 
occur because “not all possible aspects of a thought object are necessary to 
its construction, just as little as all possible aspects of a visual object are 
necessary to the unambiguous comprehension of its structure.” Duncker 
maintains, however, that the inspectional (synthetic) technique can become 
a handicap in problem solving, as the thought material is then more thoroughly 
imbued with perceptual functions, which may be a hindrance in restructuring 
problems. This observation is interesting in the light of the slight negative 
correlation of both C, and C; with synthetic reasoning, R. 


TABLE 11 
Test Vectors as Linear Combinations of Primary Vectors, A = VD 











C3 C2 P W S: CQ R H 
1. Figures .05 —.12 .03 —.10 .91 —.03 —.10 —.02 
2. Cards —.10 .12 —.04 .10 .79 .04 .08 .05 
3. Gestalt Completion .03 —.06 .238 —.06 .43 .36 .08 —.10 
4. Mutilated Words .64 .09 —.01 .03 —.03 .58 .08 .08 
5. Concealed Figures 12 .79 .28 —.07 17 .07 .01 —.07 
6. Designs .08 .39 .55 .04 .01 -—.07 —.01 —.09 
7. First Letter .03 .00 .04 .65 .10 —.08 —.08 | 
8. Suffixes 05 —.04 —.03 77) —.138 .05 .04. —.038 
9. Writing Phrase , .10 —.04 .03 .09 —.06 .03 .05 13 
10. X’s .02 .09 —.08 —.04 .28 —.11 —.35 .29 
11. Identical Forms — .07 01 .59 .07 .34 —.03 at .O1 
12. Identical Numbers 54 —.07 65 —.03 —.17 .07 .01 .02 
13. Number Series .32 .40 .04 .03 .01 .00 438 —.07 
14. Letter Series .56 .03 .31 —.01 .10 —.08 .55 .08 
15. Figure Classification .05 .58 .04 .04 .07 —.05 .46 —.05 
16. False Premises .07 .58 —.09 .01 —.14 .00 .40 RC 
17. Hidden Words 57 —.04 .62 —.04 .04 —.11 01 .08 
18. Incomplete Words .81 04 —.14 .09 —.04 .34 .06 .06 
19. Anagrams 91 —.12 .08 —.03 .O1 .09 .01 —.01 
20. Scrambled Words .88 —.06 .07 .07 .038 —.05 —.05 —.21 
21. Copying .00 61 .05 .01 .09 .32 —.01 .08 
22. Four-Letter Words .61 .09 .325 —.09 —.03 11 —.10 .15 
23. Backward Writing .25 01 .28 .3l .16 .07 —.02 —.06 
24, Hidden Pictures .03 .03 .08 —.07 .16 .62 —.26 —.03 


25. Sentences —.13 01 .08 .36 .16 .O1 .28 .48 
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“cc 


Burt (3), in describing two types of problem-solvers, writes: “‘. . . the 
characteristic procedure of the former group is explicit or analytic reasoning, 
depending on a succession of logical steps; of the latter, an implicit or syn- 
thetic apperception of what later psychologists have called Gestalten or con- 
figurations, depending on an intuitive insight which embraces the component 
aspects almost simultaneously.” 

It would seem that the C, factor is associated with the more analytical 
method of problem-solving, whereas factor R may represent the synthetic 
ability discussed by Burt, Duncker, and Yela, among others. 

The C, factor seems to correspond closely with Meili’s plasticity factor, 
where the Gottschaldt Figures, Kohs’ Blocks, problems taken from Duncker, 
and Number Series all have significant loadings (10). It is also similar to 
Factor A in Rimoldi’s study (11). 


3. Perceptual Speed, P 


Test Loadings on C Components of P 
17. Hidden Words .46 .62 
11. Identical Forms 44 .59 
12. Identical Numbers 41 .55 
6. Designs 41 .50 
22. Four-Letter Words .26 .35 


This factor corresponds with Thurstone’s perceptual speed factor, P 
(14, 15). The test with the highest loading on this factor is Hidden Words. 
This test was constructed in an effort to obtain a verbal equivalent of the 
Concealed Figures test. At the time of construction the writer hypothesized 
that the test might have its chief loading on perceptual speed, since it is 
difficult to write sentences with such strong Gestalt properties that one’s 
perception of the embedded word is hindered. It was, in fact, gratifying to find 
that Hidden Words has almost equal components of C; and P. 

The tests high on this factor require the subjects to find a given stimulus 
among distracters, which do not have strong Gestalt properties that distort 
or conceal the desired stimulus. 


4. Word Fluency, W 


Test Loadings on D Components of W 
8. Suffixes .53 BE | 
7. First Letter 45 65 
25. Sentences .25 36 


The three tests with loadings on this factor define one of the most clear- 
cut hyperplanes in the simple structure. The tests Suffixes and First Letter 
were introduced into the present battery as key tests for the word fluency 
factor, W. Sentences has an obvious “fluency-with-words” component, al- 
though it does not require the subject to think of isolated words fitting some 
restriction, which is how word fluency has been interpreted previously (14). 














CAROL PEMBERTON 281 


The test Sentences was introduced because Meili (10), using a similar test 
and a population of children, had obtained a loading on the globalization 
factor. In our case the highest loading for this test was on speed of hand- 
writing, H. The test did have a small loading on R, which seems similar to 
Meili’s globalization factor. For adults, well versed in using words, the ability 
to induce something common to the given words does not seem to be as 
important as in the case of children. 


5. Space, S; 


Tests Loadings on FE Components of S 
1. Figures .64 91 
2. Cards .56 19 
3. Gestalt Completion .30 43 
11. Identical Forms .24 34 
10. X’s .20 .28 


The highest loadings on this factor are the two key tests for the first 
space factor, 8, . Thurstone interprets this factor as ‘‘the ability to visualize 


TABLE 12 
Correlations between the Tests and Primary Factors, R,, = FT’ 








C; Cy r W Ss: Ci R H 
1. Figures 24 sor .3l .18 81 13 .O7 —.12 
2. Cards .30 Be .30 31 87 2a 19 —.08 
3. Gestalt Completion .02 .22 51 .03 OF .58 2 —.18 
4. Mutilated Words .5e 47 .32 .40 .36 41 05 .05 
5. Concealed Figures 30 81 43 .18 .65 386 —.04 .02 
6. Designs .38 .50 On Be 46 21 —.13 .07 
7. First Letter .58 oe 19 76 .28 —.19 —.25 . 23 
8. Suffixes 49 6 07 BYP ll —.12) —.07 .04 
9, Writing Phrase : 13 .06 .18 19 —.10 —.05 —.25 43 
10. X’s .24 31 07 IS 19 +—.18 —.44 ot 
11. Identical Forms .16 Zo ta 24 08 .34 OS 06 
12. Identical Numbers 44 .19 56 .30 .18 .16 —.138 .16 
13. Number Series 49 Ra .08 .30 45 .08 oo —.17 
14. Letter Series OD oo .o2 .42 48 07 .38 —.04 
15. Figure Classification .30 56 .06 iy 48 .13 36 —.15 
16. False Premises 20 46 —.09 .138 22 .06 .25 —.038 
17. Hidden Words .60 .33 .64 .48 38 05 —. 18 .22 
1S. Incomplete Words 206 AY 1] OF ol .06 —.04 .05 
19, Anagrams Si .39 .18 iY 31 —.11 —.10 .02 
20. Scrambled Words 87 dl 09 64 4 —.25 —.13 —.138 
21. Copying .29 710 .30 ei .48 41 —.06 .12 
22. Four-Letter Words .60 245 AT 43 3 .09 —.28 .28 
23. Backward Writing .o4 .38 .44 Ob AT .11 —.09 04 
24. Hidden Pictures — .04 20 38) —.04 .24 oe —=.1] .02 
25. Sentences 1d 15 26 bd 22 .08 .08 42 
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a rigid configuration when it is moved into different positions” (19). This 
ability would certainly be called for by the tests Figures and Cards. In the 
case of Identical Forms and Gestalt Completion no movement from one posi- 
tion to another occurs. However, Identical Forms has consistently had a 
small loading on §, (1, 2, 14). The loading of Gestalt Completion on this 
factor has not occurred previously. It may be that in devising new forms of 
this test the factorial composition has been somewhat changed. 

The small loading of X’s on 8, may not have occurred by chance. Chap- 
man (4) in analyzing the MacQuarrie test found a correlation of .68 between 
the factors he identifies as spatial and motor. The highest loading on the 
spatial factor was given by the test Dotting, which requires skill similar to 
that used in making X’s rapidly. 


6. Speed of Closure, C; 


Tests Loadings on F Components of C; 
4. Mutilated Words 48 .58 
24. Hidden Pictures .39 .52 
3. Gestalt Completion 2 .36 
18. Incomplete Words .25 .34 
21. Copying .24 .32 


This is evidently the factor identified as speed of closure in previous 
studies (2, 19). Mutilated Words and Gestalt Completion were the two tests 
which defined this factor before, and Hidden Pictures and Incomplete Words 
had loadings on it. This factor is represented by tests in which the subject 
must unify a relatively unstructured perceptual field. 

Scrambled Words and Four-Letter Words, which had small loadings on 
this factor in Botzum’s study, do not appear on our C, factor. The differ- 
ence between these two tests and Mutilated Words and Incomplete Words 
seems to be that in the latter no rearrangement of parts must take place, as 
in Scrambled Words, or extraction from a background, as in Four-Letter 


Words. 


TABLE 13 
Centroid Factor Matrix, F2 








I II Ill IV V y 





Cs; .80 15 — .43 .O1 mo a5 87 
C, 65 .28 Ys — 30 —.31 ti 

P .30 31 .50 .40 14 .62 
W . 69 .12 — .34 .23 .08 67 
Si 45 .61 .09 —.27 .24 oth 
Ci —.17 54 .50 .23 — 06 63 
R —.41 .50 —.2 — .06 me 47 


H 32 — .40 31 ay .05 .39 
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TABLE 14 
Fifth-Factor Residuals in Second Order 








Cs Cy Y W Si C R H 

C; .00 .03 —.01 — .02 .00 Ol —.01 
C2 .00 — .06 01 .03 .02 — 01 .03 
P .03 — .06 =—<OL .06 .02 nue 02 
Ww —.01 O01 —.01 —.01 .00 .00 .02 
Si — .02 .03 .06 — OF — .03 .O1 — .02 
C .00 02 .02 .00 —nOS 02 .00 
R 701 —.01 —.03 .00 01 .02 .O1 
H —.01 .03 — .02 02 — .02 .00 .O1 





One might think that Hidden Pictures would require flexibility of closure 
rather than speed of closure, but this test has consistently had loadings on 
C,. It seems that it is not breaking the Gestalt formed by the large picture, 
but the sudden combination of the correct elements, which accounts for indi- 
vidual differences in Hidden Pictures. In Concealed Figures and Designs 
the subject is shown the figure which he is to locate and can therefore perform 
the task analytically, whereas in Hidden Pictures he merely knows that he is 
looking for pictures of people. Reproductions of the faces to be found could 
be given preceding each large picture, to find out whether the test would then 
have a loading on C, . 


7. Reasoning, R 


Tests Loadings on C,; Components of R 
14. Letter Series .46 .55 
15. Figure Classification .38 .46 
13. Number Series .36 43 
16. False Premises .33 .40 
25. Sentences .23 .28 
24. Hidden Pictures — .22 — .26 
10. X’s — .29 — .35 


The three tests with highest loadings on this factor are all classic tests 
of induction. As previously stated the writer prefers to differentiate reasoning 
factors in terms of the mental processes involved, rather than in terms of 
the types of problems involved. Figure Classification, Number Series, and 
False Premises each have higher components of the C, factor, and Letter 
Series a higher component of C; , than they have of R: The abilities repre- 
sented by C, and Cy, were interpreted as being analytical in nature. It may 
be that the synthetic apperception of the solution may be represented in the 
present factor. Letter Series would appear to be the test which lends itself 
most easily to this form of solution. Some subjects perform this test by 
singing the series sub-vocally, which is a method of solution that depends on 
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TABLE 15 
Oblique Factor Matrix, V2 

















AA BB CC DD EE* 
C3 15 .03 — .05 57 —.138 
Cz .65 .00 .02 — .02 — iol 
P — .01 =.07 61 15 .09 
W — .05 —.01 .07 .59 07 
8; .55 OF .02 01 .25 
C, .03 .30 .62 — 208 —w13 
R — .04 65 .03 .03 .00 
H — .02 — .59 .09 — .05 13 


* Residual 


perceiving the series as a Gestalt, rather than analyzing the principle upon 
which it was built. It has already been mentioned that some of the variance 
of Sentences is accounted for by the ability to induce the common elements 
which permit the given words to be combined into a meaningful sentence. 
Our interpretation of this factor as reasoning “‘by closure of the unfinished 
configuration,”’ to use Yela’s words (20), seems to be justified by the inter- 
correlations of this factor with the others, since its highest positive correlation 
is with speed of closure (.24). This gives some corroborative evidence for 


TABLE 16 
Transformation Matrix, A» 


AA BB CC DD EE 
I 34 — 33 .02 33 09 
II .22 86 1] .23 —.12 
IT] 303 39 43 — .56 OS 
r\ 86 ~ 10 75 70 —.14 
\ 00 .00 — .30 —.17 QS 

TABLE 17 
Reference Vector Cosines, Cs 

AA BB CC DD FD 
AA L.O1 .03 4) — .62 .15 
BB .03 1.0) 10 24 —.15 
$+ — 4] .10 1.01 44 — 4] 
DD — 62 24 14 .99 — 31 
| DF Di 15 —.15 —.4] —.3 1.01 
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Thurstone’s supposition that “the first closure factor might be associated with 
inductive thinking,” although the association is not as strong as the association 
of the second closure factor with analytical thinking. 

This is the only factor in which there are negative loadings to account for. 
X’s was the last test in the battery, and the subjects were aware of this. The 
brighter subjects may have been bored by this simple task, whereas the less 
apt subjects may have felt this was ‘‘positively their last chance” to make a 
good score. The negative loading of Hidden Pictures may also be due to 
motivational factors. 

Factor R is similar to Factor G in Rimoldi’s study (11), on which Figure 
Classification, Letter Series, and Number Series all had loadings. Referring 
to the tests with loadings on G, Rimoldi writes: ‘‘. . . In the easy items the 
analytical activity is limited and the grouping of the letters and numbers is 
rather obvious. As soon as the problems increase in difficulty the grouping 
becomes less obvious, and the subject has to discover the structure characteriz- 
ing each item.” It is probably the easier items which account for the loadings 
of these tests on R, but further research is needed to verify this. 


8. Speed of Handwriting, H 


Tests Loadings on H Components of H 
(Reference Vector) (Factor) 
9. Writing Phrase .64 .73 
25. Sentences .42 .48 
10. X’s .25 .29 


This seems to be the same factor as Bechtoldt’s (1) factor H, which he 
identifies as “the ability to operate hand-finger neuromuscular response 
mechanisms at a high rate of speed for short periods of time.” As we only have 
tests involving handwriting represented here, we have identified the factor by 
the simpler term “speed of handwriting.” It was gratifying to find that this 
factor had only one positive correlation of any size (.20), this being with 
perceptual speed. Individual differences in ability to write quickly therefore 
did not appear to affect the scores of most of the paper-and-pencil tests used 
in this battery. 


VII. Interpretation of the Second-Order Factors 


With only eight primary factors it is not possible to determine the second- 
order factors with much confidence. The interpretation of these factors is 
therefore highly tentative. 


Factor AA 


Flexibility of Closure .65 
Space .55 








286 PSYCHOMETRIKA 


It must be remembered that our flexibility of closure factor had three 
reasoning tests with high loadings on it. Bearing this in mind, it becomes 
apparent that this factor is very similar to the second-order factor a, isolated 
by Botzum, on which space, deduction, induction, and flexibility of closure 
all had loadings over .60. 

In the mechanical aptitude study, although a second-order analysis was 
not performed, it seems likely that inductive reasoning, space, and flexibility 
of closure would come up on one second-order factor, since I and C; correlate 
.63; I and S, , .38; and S, and C, , .53. 

The space primary has loadings on both this factor and factor BB. This 
also occurred in Botzum’s study. He points out that one can perform the space 
tests in two ways. One can either analytically reason out which way the figure 
would be facing, when turned in the plane of the paper; or one can actually 
imagine the rotation of the figure. It is our hypothesis that the analytic pro- 
cedure in problem-solving is what is represented in Factor AA. 


Factor BB 
Reasoning .65 
Space 37 
Speed of Closure .30 
Speed of Handwriting — .59 


This factor seems to represent the ability to solve problems synthetically. 
We tentatively interpreted our reasoning factor R as the ability to synthesize, 
and above we mentioned the possibility of solving the space tests by two 
different methods, one analytic, the other synthetic. The synthetic element 
in speed of closure is obvious. Both Jones (8) and Rimoldi (11) have reported 
second-order factors which they interpret as synthetic ability. 

The negative loading of speed of handwriting on this factor is difficult 
to account for. In Bechtoldt’s study speed of handwriting correlated —.22 
with Factor G, which seems to correspond to our C, factor. Future studies 
will establish whether this negative relationship occurs consistently. 


Factor CC 
Speed of Closure .62 
Perceptual Speed 61 


This factor seems to be speed of perception. It is similar to factor F 
found in the factorial study of perception (17). It is not a more generalized 
speed factor, since speed of handwriting does not have a loading on it. Simi- 
larly, Thurstone’s F was clearly differentiated from reaction time. Dispersed 
attention seems to be required in both the speed of closure and perceptual 
speed tests. 

Factor DD 
Word Fluency .59 
Verbal Closure .57 





e 
1S 


—— — _— ae 
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This factor is interpreted as the ability to think of words rapidly which 
fit certain formal requirements, such as beginning with a certain letter, ending 
with a certain suffix, containing certain given letters, etc. If our interpre- 
tation of the verbal closure factor was correct, one would expect that C; would 
have a loading on the second-order factor AA as well as DD. However, its 
loading on AA is only .15. Due to the indeterminancy of the second-order 
structure, this does not necessarily invalidate our interpretation of C; . 
Furthermore, the cosine of the angle between AA and DD is —.62; therefore a 
positive correlation between the second-order primaries corresponding to AA 
and DD could be predicted. This is probably due to the flexibility required 
by all the tests with loadings on these two second-order factors. 

Factor EE has been left without interpretation as it appears to be a 
residual factor. 


VIII. Discussion 


The aim of this study was to ascertain whether abilities on speed of closure 
and flexibility of closure tests would generalize to tasks requiring higher 
cognitive functions. It yielded strong evidence that flexibility of closure is 
associated with analytical reasoning. The association is so close that only one 
factor appears on which all the flexibility of closure tests and three of the 
reasoning tests have loadings. Correlated .54 with this factor is one in which 
rearrangement of letters or numbers to form new Gestalts, or the picking out of 
Gestalts from a distracting background, is required in all the tests. This 
indicates that there is a connection between perceptual flexibility, the flexi- 
bility required to solve analytical reasoning problems, and the flexibility 
needed to solve problems utilizing highly practiced symbols but where meaning 
is not important. 

The evidence for the generalization of speed of closure in the higher 
cognitive domain is not so clear-cut as for flexibility of closure. It is only in 
the second order that we find a coming together of the speed of closure and 
reasoning tests, but this second-order factor does seem to indicate that both 
perceptual and reasoning problems can be solved by a rapid synthetic process. 
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SOME EXAMPLES OF MULTIVARIATE ANALYSIS 
IN EDUCATIONAL AND PSYCHOLOGICAL RESEARCH 


Frep. T. TyLer 


UNIVERSITY OF CALIFORNIA 


This paper provides the formulas necessary for testing the significance 
of the differences between mean values of different multivariate normal popu- 
lations by Hotelling’s generalization of ‘‘Student’s” ratio. It also indicates 
the methods, proposed by Mahalanobis, and by Rao, of classifying different 
multivariate populations and individuals. The methods are illustrated by 
means of personality test data obtained from students preparing for the 
General Elementary and the General Secondary Teaching Credentials. 


The multiple regression equation serves two primary purposes: It provides 
a method for estimating the value of a battery of tests in predicting some 
criterion, and it furnishes a basis for determining the most appropriate weights 
to give each subtest in the battery. Problems which employ multiple regres- 
sion equations, then, are concerned with the prediction of one variable from a 
series of other variables, all variables being expressed in quantitative terms. 
Analogous to the problem of prediction is that of classification of individuals 
where the criterion is more qualitative than numerical in nature. Thus, we 
may be interested in differentiating between two or more groups of individuals 
rather than in predicting the numerical value of the dependent variable for 
each individual. Fisher (1) proposed the discriminant function as a suitable 
technique for this purpose. The discriminant function provides material for 
answering the question: ‘‘What is the best system of weights to apply to the 
various independent variables in order to produce a maximum separation 
between the groups?” As Garrett (4) points out, this becomes a multiple 
regression problem if the numerical value of the dependent variables for each 
individual is known. 

There is another more general type of question that might be asked: 
(1) “Could these two groups have arisen from the same population?” This 
is clearly a question of the allocation of groups to a population rather than of 
prediction. The problem propounded is not concerned with maximizing the 
difference between groups, but is simply concerned with the similarity of the 
two groups; nor is the question related to the prediction of the numerical 
value of a dependent variable. Having obtained a negative answer to this 
particular question, the investigator may proceed to ask two other questions: 
(2) “Can a given individual be classified into either of the two groups under 
study?” (3) “If so, into which group is he most properly classified?” 
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Hotelling (6) has proposed a method of answering the original question: 
Are these two groups significantly different from one another? Mahalanobis 
(9) and Rao (11, 12, 13) have suggested procedures for answering the other 
two questions. This paper will describe and illustrate the methods. A com- 
plete explanation is not possible within the limits of the paper; certain features 
of the development must be assumed. The mathematical development of 
these tests is available in various sources. Analogy of the technique with the 
t-ratio for small samples will be pointed out. 


Problem I. To Test the Significance of the Difference Between the Mean Values 
of Multivariate Normal Populations. 


Typically this problem has been attacked by determining the significance 
of the difference between the means of the two groups on each variable suc- 
cessively. Such a procedure does not make maximum use of the available 
data since it does not consider the inter-correlations of the variables in each 
of the groups being studied. The é-test considers the variance in each of the 
variables but does not make use of the covariances of the variables. Hotelling 
(6) proposed the 7-test to study the significance of the difference between two 
multivariate normal populations where it can be assumed that the two popula- 
tions have the same dispersion matrix, i.e., have equal variances and co- 
variances. 

Suppose we give the same r tests to two groups, one group having N, 
subjects and the other N, . 


Let X; (i = 1,2, --- , 7) be the random variable defined by the ith test 


in the first group, 
Z; (¢ = 1,2, ---,7) be the random variable defined by the 7th test 


in the second group, 

X ;, be the score of the kth individual in the first group on the ith 
test, 

Z;, be the score of the kth individual in the second group on the ith 


test, 
X; be the sample mean of the 7th test in the first group, and 


Z; be the sample mean of the 7th test in the second group. 


Assume that the dispersion matrices for these tests from the two samples are 
equal. Then the best (unbiased) estimates of the sample variance for the 


7th test is 


=1 
2 _— Ns _—— 

| nu ~1R<+ Sw - wai], 
=] k=1 


where n = N,+ N, — 2, and7 = 1, 2,---,7. 
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Similarly, the unbiased estimate of the covariance, 7 ~ 7, becomes 


1] 5 (Xu — X)(Xpn — X) + > (Zin — ZZ - Z) | 


a;; = a; 


Ae 


Ne aan e Ne prgeee 

= 1] X pX jp — NX,X; + ¥ tuts. -— NEE, | 
k=1 k=1 

where n = N, + N, — 2, andi,j = 1,2, ++: , 7,4 47. 

Let 





d, = —t = = (X, - ~ 2) fg. 
¥ nt WV. 
The dispersion matrix has the determinant: 
Qi. Gig **° Gir 


Mai G22 *°° Aer 








Ary a2 rie Orr 


Define A;; as the co-factor of a;; in the determinant a. 

The traditional procedure for testing the significance of the difference 
between two means in educational and psychological research has been by 
means of the é-test for single tests, 

X, —Z,; rN, 
Va, VN.+N, 





i, = 


and with NV, + N, — 2 degrees of freedom. 
Hotelling’s 7-test for testing the significance of the difference between 
two multivariate normal populations is 


‘ b> > Avid, | 
t=1 j=1 


which reduces to ¢; when we consider only the 7th test. 
In order to determine the significance of the computed value of 7”, it is 


necessary to compute 


t= ae 
142 


n 
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The probability that a value as large as or larger than 7” could occur by chance 
can be determined from Pearson’s tables of the Incomplete Beta Function, 
I,(p, q), where p = (n — r+ 1)/2 and q = r/2 (10). The table indicates the 
probability that the obtained value of 7 will be exceeded by chance if in 
fact there is only one population. 

The significance of T” may also be determined from the F tables, 


with r degrees of freedom for n, (the larger variance) and (n — r + 1) degrees 
of freedom for nz (the smaller variance). 


Problem II. The Classification of an Individual into Either of Two Populations. 


If we reject the hypothesis that two samples are members of a single 
population, it sometimes happens that we wish to know whether a certain 
individual can be classified into either of the two specified populations. The 
procedure available for such a problem is an extension of Problem I to consider 
the case where three groups are under consideration, one group consisting of the 
individual to be classified (13). As in the preceding problem, it is necessary to 
set up the dispersion matrix of the unbiased estimate of the variances and 
covariances. 


Let Y, (¢ = 1, 2, 3, --- , r) be the random variable defined by the 
ith test in the third group, 
Y;, be the score of the kth individual in the third group on the 
__ tth test, and 
Y; be the sample mean of the 7th test in the third group. 


Then the unbiased estimate of the sample variance for the 7th test is 
ii a... =e < = — voaees 
Qi: = n | - (X an x7 + > (Zix <a Zz. + 2. (Yur —- r »*|, 
k=1 k=1 k=1 
where n = N,+ N,+ N, — 3, andi = 1,2, ---,7r. 


Similarly the unbiased estimate of the covariance, 7 ¥ j, is 


17 2 . _ P ~ 
a;; = a; = n Fe (Xix — X MX jx a X ;) 


+ 2 ts ~ Rie, ~ E+ 2 e. ~ Tavs, YI, 


where n = N. + N,4+ N, — 3, andi,j = 1,2,3--- 7,1 Fj. = 

In the present case N, is 1, so that Do?2, (Yi. — Y:)? and )o?, (Ya — Y;)- 
(Y;. — Y;) each has the numerical value of zero. Consequently the deter- 
minant in this situation has the same value as the determinant found for 
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Problem I; similarly the co-factors will be numerically equal to those in the 
preceding problem. 

It is now necessary to test the significance of the difference between the 
scores of an individual and each of the corresponding mean scores of the two 
populations with which the individual is being compared. 


Let d; = (Y; — X,) — (Z; — Y,), and 1/N = 4/N, + 1/N, + 1/N,. 
Define 


v=: |a( oe ud.d;) 


i=1 j=1 


and 








: (V) 


pa ([%s+N tN. =2—7] 


with n, = (r) d.f. for the larger variance and n. = (V, + N, + N, — 2 — 1) 
d.f. for the smaller variance. 

The F tables indicate the level of significance with which an individual 
can be classified into one of the two populations under consideration. 


Problem III. The Classification of an Individual into One of Two Populations. 


Having determined that an individual can be classified into either of 
two populations, it may then be desirable to determine into which population 
he is most appropriately placed. This involves the computation of the 
generalized distance function, D*, from the values of the co-factors and deter- 
minant already available from the calculations in Problem II. Let the popu- 
lations under consideration be II, , Il, , and II, , where II, is the individual 
to be classified. The generalized distance function is computed for I, and 
II, , and for I, and II, . 

Letd; = xX, — Y; 


and ee = ‘ | _ A;,djd; 


t=1 j=1 4 


Similarly, let d; = } a _ Y, 


Zz 2, Aig, 


1 
and De. =" 
QLin1 j=1 








The individual is then classified into that population for which D’ is smaller. 


Illustrative Problems 


Problem I. 
The Heston Personal Adjustment Inventory was given to 145 female 
students training for the elementary teacher’s credential at the University of 
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California and 100 women training for the general secondary credential. 
Three of the subtests of the Heston Inventory were selected for study, viz., 
Emotional Stability, Confidence, and Personal Relations. The hypothesis 
to be tested is that the two groups arose from the same population. 

The statistics required for the calculation of T’ are shown in Table 1, 
which includes the sums, the sums of squares, and the sums of cross-products 
for each group. 


TABLE 1 
Sums, Sums of Squares, and Sums of Cross-products 








>Xu = 3267 Zu = 4897 2 XuXx = 104027 
DXx = 3076 Zu = 4194 ZXuXu = 99335 
Xx = 2979 Zu = 4182 xX XunXs = 93606 
> X?, = 111609 > Z?, = 143269 2 ZuZx = 134181 
> X2, = 99332 > Z2, = 129748 2 ZuZs = 131567 
> X?2, = 90803 2 Z3, = 124768 2 ZZ, = 125150 
N, = 100 N, = 145 





The computation of T’ from the above data is next illustrated. 
a. Determine the values of d; : 
d, = 18.046932; d, = 14.123455; d; = 7.297829. 
b. Compute the numerical values of the estimates of variances and co- 
variances: 


a,, = 60.945990 Qoo = 54.133356 
Qi. = 43.356566 23 = 25.354795 
Q\3 = 27.829322 Q33 = 25.563546 


c. The expression for the determinant becomes: 
| 60.945990  43.356566 —-27.829322 | 
a = | 43.356566 54.133356 25.354795 | 
| 27.829322 25.354795 25.563546 | 
d. Calculate the values of the co-factors: 


A,, = 740.974907 As = 783.524456 
Ay. = 402.740815 As, = 338.689246 
Ais = —407.197752 Ass = 1419.419158 


e. Compute the value of the determinant: 
a = 16365.953188 

f. The value of 7° may now be determined: 
T”’ = 39.171612 


. Calculate the probability that a T° of this magnitude or larger could have 
arisen by chance if the two groups had arisen from the same population: 


1 
1+ 


02 


x = .861178 
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Enter Pearson’s Incomplete Beta Function tables with p = 120.5, g = 1.5. 
If Pearson’s tables are not available, it is possible to determine the significance 
of T’ by means of the F tables: 

F = 12.9497, with n, = 3, and n, = 241. 
An F of 3.88, with n, = 3, n. = 200, is significant at the .01 level. 


Problem II. 


Can the individual Y, = 35, Y. = 29, Y; = 28 be classified into either 
of the two populations of Problem I? 
a. Calculation of the values of d; : 
d, = 7.005862 


d, = —1.684138 
d; = —2.631379 
b. Compute the value of N: 
N = .248948 
c. Determine the value of V: 
V = .774669 
d. Calculate the value of the variance ratio (F): 
F = 62.2317 


e. Determine the significance of F = 62.2317, with n, = 3, and n, = 241. 
This value of F is significant at better than the .01 level. We conclude that 
the individual can be classified into one of these two populations. 


Problem ITI. 


Into which population of Problem I is the individual Y, = 35, Y, = 29, 
Y; = 28 best classified? 
a. Determine the values of d, for II, , II, : 


d, = — 2.33 
a= LG 
d,= 1.79 


b. Compute D?,,,, 
D...2, = 30800 
c. Calculate the values of d, for IT, , I, : 
d, = —4.675862 
d. = — .075862 
ds 841379 
d. Compute Dj,.,, : 
Die, = 1.962147 
e. Since Df,,,, is less than Df,,,, we conclude that the individual (Y, = 35, 
Y, = 29, Y; = 28) is best classified with II, . 


Il 


Concluding Statement 


The purpose of the present paper was to present procedures for answering 
three types of questions: 
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(1) Could two groups of subjects have arisen from the same population? 
In case the answer to the preceding question is negative, a second question 


arises: 
(2) Is it possible to classify a given individual into either of the two 


populations mentioned in the first question? 

If an affirmative answer is obtained, then a third question arises: 

(3) Into which of the two populations is the individual best classified? 

The formulas necessary to the solution of each question have been pre- 
sented. An application of these methods of multivariate analysis has been 
made to a set of data about students preparing for the elementary and the 
secondary teaching credential. 
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NOTES ON A PROBLEM OF MULTIPLE CLASSIFICATION* 


FrEpDERIc M. Lorp 


EDUCATIONAL TESTING SERVICE 


A solution is developed in implicit form for the problem of assigning N 
— n jobs, the proportion of men to be assigned to each job being specified 
in advance. 


Suppose that it is desired to assign N men to n jobs, the proportion of men 
to be assigned to each job being specified in advance. It is desired to maximize 
the average weighted productivity of the men, the productivity of each man 
being weighted according to the importance of the job to which he is assigned. 
It is assumed that the productivity of each man for each job is known in 
advance and can be used as a basis for assignment. If x;, is the productivity 
of man a for job 2, we can indicate the productivity of all men assigned to job 
1 by by Liq. The quantity to be maximized is then 


Q = Wi>*ie + W2>* 220 + ees + Wrd>*2na ’ (1) 


where w, is the weight assigned to job 7. 

A solution to an almost identical problem, assuming all weights to be 
unity, has been given by Hubert E. Brogden (1); these and related problems 
have more recently been treated by Thorndike (2); Votaw (3) has quite 
recently made important contributions toward a rapidly converging successive 
approximation method for the practical solution of such problems.t The 
present paper is primarily concerned with developing in analytic form an 
implicit solution that is effectively the same as Brogden’s; it is not immediately 
concerned with the problem of obtaining a practical solution by successive 
approximations. 


The Two-Dimensional Case 


Consider first the case when n = 2. The contribution of any individual 
to Q will be w,2,, if he is assigned to job 1, w.%2, if he is assigned to job 2. 
The differential effect of the job assignment on Q is in this case w,%,, — W2%2.. 
All individuals for whom the quantity w,7,, — wW.%2, has a given value are 
interchangeable with each other in their effect upon Q, and consequently are 

*The author wishes to thank Dr. Hubert Brogden and Dr. Paul Horst for their helpful 
discussion and criticism. 


Dr. Paul S. Dwyer has recently developed important practical methods of solution, 
as yet unpublished. 
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interchangeable with each other for purposes of job assignment. Furthermore, 
if certain individuals characterized by some specified value of w,2%1, — We%0q 
are properly assigned to job 1, then all individuals with higher values must 
also be assigned to job 1. The same relation obtains between w.2,, — W,%, 
and assignment to job 2. 

Suppose a scatter diagram is plotted with x, and 2, as axes, each indi- 
vidual being represented by a point corresponding to his values of x, and 2, . 
The reasoning just given shows that the optimum assignment of people to 
jobs corresponds to the division of the scatter diagram into two regions by the 
line w,x,; = Wt, + k, where k is a constant to be determined so that the 
required proportions of individuals fall in the two regions created. 

We thus have the result for n = 2 that the optimum assignment cor- 
responds to a region bounded by a straight line with a slope of w,/w,. This 
conclusion holds irrespective of the shape of the bivariate frequency distribu- 
tion represented by the scatter diagram. The intercept of the line must be 
determined so that the proportions of cases cut off by the line are equal to the 
predetermined proportions of people to be assigned to the two jobs. 


The Three-Dimensional Case 


Let us next consider the case where n = 3. Using 2, , x. , and 2; as axes, 
a three-dimensional scatter plot may be prepared representing the data. The 
region of space containing the individuals to be assigned to job 1 will neces- 
sarily include all positive values on the x, axis above a certain point. Consider 
the surface bounding this region (region 1) from the region containing the 
individuals to be assigned to job 2 (region 2). 

Suppose that the desired regions have already been set up. There will be 
of necessity certain individuals who could equally well be assigned to jobs 
1 or 2, but who should not be assigned to job 3. Such individuals lie on the 
boundary surface between jobs 1 and 2. For such people, the xz; score can 
have no effect on the decision as to whether they should be assigned to job 
1 or to job 2. Consequently, the boundary between region 1 and region 2 is 
parallel to the x, axis. By the same reasoning used for the case where n = 2, 
it follows that this boundary must be represented by the equation w,z, = 
WL + k,. where k,. is a constant to be determined so as to obtain the proper 
proportion of people in each region. Similarly the boundaries between regions 
1 and 3 and between regions 2 and 3 are respectively w,7, = w 3%; + k,3 and 
WL, = W3%; + k.;. In the present case these three equations define 3 planes, 
each of which is functionally independent of one of the variables and conse- 
quently parallel to the corresponding coordinate axis. 

Let us see if any further conditions should be imposed on these planes. 
Let p,; be the proportion of cases to be assigned to job 7. Since >>” p; = 1, 
only n — 1 = 2 of the values of p; can be determined arbitrarily. Since the 
values of the k’s must be adjusted so that each region contains the proper 
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proportion of cases, not more than one restriction can be imposed on the 
three k’s. It will now be shown that one restriction must be imposed. 
Consider the augmented matrix representing the three planes: 


| Ww, —W2 0 Kxs | 
| 
| W, 0 —W3 kes 
| 0 W,2 —W; kos | : 


If this matrix has a rank of 3, the three planes will intersect in only one point, 
and hence will define 2” = 8 regions. Since we require only three regions, we 
would have to select some combination of these eight regions that would form 
the three regions required. Furthermore, symmetry requires that each of the 
three regions be defined with respect to the three coordinate axes in some way 
that is symmetrical, as far as the axes are concerned, with the definitions 
for the other two regions—e.g., if one boundary is a half-plane, all boundaries 
must be half-planes; if two half-planes intersect each other in a line, all half- 
planes must intersect all others in a line, ete. Since it is manifestly impossible 
to form three regions from the eight available so as to meet this condition, it 
follows that the rank of the matrix must be less than 3. 

This same conclusion may be reached via an alternative argument. There 
must be, at least theoretically, certain individuals who could be assigned to 
any one of the three jobs without changing the value of Q—such individuals 
will lie at the mutual intersection of the three boundary planes. Moreover, 
such individuals may vary in at least one dimension of our coordinate space, 
since their contribution to Q (irrespective of the job to which they are assigned) 
may vary in one dimension, i.e., may be large, small, or intermediate in value. 
These cases therefore cannot all be located at a single point in our coordinate 
space, but must be distributed at least over a line. Since all these cases lie at 
a place where all three-regions are in simultaneous contact, it follows that the 
planes bounding the regions must all have at least one line in common. This 
requires that the rank of the matrix given must be less than 3. 

Since the w’s are predetermined, and since one and only one condition can 
be imposed on the k’s, it follows that the rank of the matrix is exactly 2, 
and hence that the planes have only a straight line in common. 

In summary, for the case when n = 3, the boundaries of the three regions 
are the three planes represented by the matrix given, with the additional 
condition that the k’s must be so chosen as to give the matrix a rank of 2. 
Kach plane is parallel to one coordinate axis and intersects the other two 
planes in a straight line. The slopes of the planes are determined by the pre- 
assigned weights and are independent of the distribution of cases in the scatter 
plot. The intercepts of the planes must be determined so that the planes cut 
off the desired proportion of the cases. 

Since three planes intersecting in the same straight line define six regions, 








300 PSYCHOMETRIKA 


we must form our three regions from these six. If we denote the plane that 
separates region 7 from region j by b;,; , we find, for example, that there are at 
least two separate regions lying between b,. and b,; . We must choose as 
region 1 the one of these that includes the end of the x, coordinate axis repre- 
senting large positive values of xz, . This is equivalent to saying that the 
boundaries of the three regions are three half-planes, each terminating in the 
same straight line that is its common intersection with the other two half- 
planes. 


The n-Dimensional Case 


These conclusions may readily be extended to any number of dimensions. 
In n dimensions, each of the n regions will be in contact with each of the other 
regions. The boundary between two such regions will be the (n -- 1)-space 
represented by the equation w,z; = w,;z; + k;; . This boundary is parallel 
to all coordinate axes except x; and z;. There will be n(n — 1) such bound- 
aries, corresponding to the 3n(n — 1) possible pairs of regions. All the 
boundaries will have as their mutual intersection a single straight line. The 
rank of the augmented matrix representing the boundaries will be n — 1. 

Since all (n — 1)-spaces acting as boundaries pass through the same 
straight line, it is possible to make a translation such that they will all pass 
through the origin. The equations for the boundaries can thus be written 
w(x; — b,) = w,(x; — b;) or wx, = wiz; + a, — a; , where the a’s (or b’s) 
are constants replacing the k’s and determining the intercepts of the (n — 1)- 
spaces. In all the preceding formulations k;; may simply be replaced by 
a, — a;. With this formulation it is seen that we need determine only the n 
unknown constants a, , a, , :*: , a, instead of the (2) values of ky, , kis , 

- Ke ; kos 9 9 Rictes 
For n = 4, for example, the boundaries are represented by 








| w, —W, 0 0 a-—-a | 

| Ww, 0 —Ws 0 a-—-a4, 

| Ww, 0 0 —W, A,— a 

| 0 W, —Ws; QO Q@— as; 
| 0 W2 QO -—-W &—-M% 

| 0 0 W3 —Wy G3 — a, 





where the a’s are restricted by the fact that the rank of the matrix must be 3. 


Formulation in Terms of Definite Integrals 
If the form of the multivariate scatter plot can be specified analytically 
by the continuous frequency function f(z, , 2 , «+: , Z,), the proportion of 
cases in region 7 can be expressed as a multiple integral, as follows: 
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a~ (wizi-—aitan)/wn (wiri—aitan—1)/wn-1 
—c 


J —-aod —-@ 


p (wiri-aytaiti)/witn (wizi—agtai—1)/wi-a 
/ -o@ J —-a@ 
(witi-aitar)/wi 
ee [ T(r » Lay °° * Be) Ot, Ely 2 ** Gey, Atia, °° * dt, dz; . 
If n = 4, the simultaneous equations to be solved for the a’s are 
ia) a (witi1—a,+G4)/we (witi—ai+a3)/ws (wiz. -G1+G9)/ws 
a=] [ I 
J-@ J-o@ J —@ -@2 
© Sle, 5 Ley Le » Xa) Uy dt, Gt, Az, , 
aw (wa%——Gg+01)/w1 (wata—-Gatag)/we (werg—Qgt+a3)/ws 
n=] | [ [ 
J-—-w /-@ “-—-@ J —w 
, f(x » T2, U3, Xs) dx; dx, dx, dx, : 
(3) 


a) (wsata—A3+a9)/we (wstas—-a3+0;)/wy (wsrts—Gat+a4)/we 


co) J—@ J-@ 


se f(a ’ V2 4 3 ’ 2.) dx, dx, ax, dx; ) 


aco (werTe—agt+as)/ws (wete—Ggtag)/ws (were—aetai)/wi 
« eo 


Jd -o@ - —o /—@ 
« I(2, , Xe , Xa, Ze) G2, Ady dx, Ax, . 


The reasoning by which these integrals are written down may be illus- 
trated by the first multiple integral given for the case when n = 4. We wish 
to find the frequency in region 1, which is known to be bounded by the three 
hypersurfaces w.%. = W;2%, — A, + G2, W3%3 = WT, — A, + G3, and wy, = 
W,2, — a, + a, . Suppose we wish to integrate first with respect to x, . 
The last two of the hypersurfaces listed are parallel to the x, axis, so they 
cannot serve as limits of integration and may be left out of consideration. 
Since the hypersurfaces can provide only one limit of integration, the other 
limit must be either + or —©, but all points for which x, = +© are 
included in region 2. Hence the limits of integration with respect to x, are 
—o and (w,2, — a, + a)/w,. Similar reasoning will give us the limits of 
integration with respect to x, and 2,. When these three integrations have 
been accomplished, there remains only one variable in our integrand—all the 
frequencies have been summed and expressed as a function of a single variable, 
x,. Since, in the general case, there will be some finite frequency correspond- 
ing to every possible value of x, , we must integrate from — © to + © with 
respect to this variable. The other multiple integrals given may be written 
down by similar lines of reasoning. 
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Since when n = 4 the a’s must be so determined that the matrix of the 
boundary surfaces has a rank of 3, and since >> p, = 1, any three of the inte- 
gral equations given is sufficient to determine the a’s. In practice, the solution 
of these equations will be extremely difficult in many cases. If f(x, , 2 , 
-++ ,2,) is taken as the normal multivariate function, the necessary expressions 
at present available for the multiple integrals are too cumbersome to be of 
practical use. Unless simpler expressions can be found, the iterative methods 
developed by Brogden and Votaw will probably be found to be the most 
satisfactory method for handling actual numerical problems. 


The Case where Some Individuals Are to be Rejected 


We may consider briefly the additional case where pp of the individuals 


Xo 






ASSIGN {TO JOB 2 








ASSIGN / TO JOB i 












FIGURE 1 
Illustrative Assignment Regions and Rejection Region for a Three-job Problem 
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are not to be assigned to any job, but are to be rejected entirely. Now if 
individuals in region 7 are to be rejected, they must be rejected solely on 
the basis of x; , irrespective of their score on other variables. Otherwise 
some people who were included in region 2 would have lower values of x; than 
those of some people who were rejected. Consequently, some hyperplane of 
the form 2; = c; , where c, is an unknown constant, must be the boundary 
(bio) between region 7 and the rejection region. 

Next consider an individual who lies exactly on the boundary between 
region 7 and region j—this individual may equally well be assigned to either 
job 7 or job 7. If this individual has a low value of z; , he may simultaneously 
lie exactly on the boundary between region 7 and the rejection region, in 
which case he also may equally well be assigned to job 7 or rejected entirely. 
It follows that this individual, and all similar individuals, may equally well 
be rejected or assigned to job 7 and that consequently they all lie exactly on 
the boundary between region j and the rejection region. We have thus proved 
that the intersection of b;) with b;; coincides with the intersection of b;) with 
b;; . Since all boundaries between jobs intersect in a single straight line, the 
boundaries of the rejection region must intersect each other in a point lying 
on this line. 

The rejection region is thus a rectangular region each of whose sides is 
parallel to n — 1 of the coordinate axes. It obviously includes all large 
negative values of all the variables. (In practice negative values of 2; may 
not occur, but we need not exclude the possibility of their occurrence in our 
theoretical discussion.) The intersection of any two sides of this region coin- 
cides with their intersection with one of the boundaries between jobs. This 
may be visualized for the three-dimensional case as follows (see Figure 1): 
The rejection region is a rectangular solid containing all large negative values 
of all variables. Only the three upper faces of this rectangular solid can 
be visualized, since the other three faces are at — ©. Each face is perpen- 
dicular to one of the coordinate axes. An oblique line having a positive 
slope with respect to all axes extends in a positive direction from the corner 
of the rectangular solid. The boundaries separating the three jobs are three 
oblique planes each of which joins this line with one of the edges of the rec- 
tangular solid and extends to © in the remaining directions. 

The analytic equations corresponding to this situation may be readily 
written down. For the general case the integrals will be the same as given 
before in equation 2, with the exception that the lower limit of the first 
integral sign will be c; instead of —. In addition to the n equations of 
the type given, there will be an (n + 1)-th equation, as follows: 


Do = [ [- - am fli 5 Va *** » Be) Ue, -+* dy dy, . (4) 


In addition to the restrictions previously imposed on the a’s, we now 
must impose n — | restrictions on the c’s corresponding to the fact that all 
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the rejection-region boundaries intersect in a single point through which all 
the other boundaries pass. These restrictions may be written down as follows: 
Wl; — Wiel; = a; — a; (t,j7 = 1, +--+ ,n). (5) 


Only n — 1 of the equations in (5) are independent. 
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The rolling totals method of computing sums, sums of squares, and sums 
of cross-products appears to have several advantages over usual methods, in 
that it saves time and requires less equipment. Only an IBM sorter and a 
tabulator equipped with Card Cycle Total Transfer Device are used. The 
method provides an immediate independent visual check on the accuracy of 
the sums of cross-products of each successive variable as it isrun. Since con- 
trolling is done by sorting, the necessity for re-wiring after each run on the 
tabulator is eliminated. The wiring, illustrated by a diagram, is simple and 
straightforward. 


Introduction 


A great deal of time is consumed in obtaining the sums, sums of squares, 
and sums of cross-products before correlational analysis between measures 
can be computed. A review of the literature indicates that there are several 
methods by which the total of products or squares is obtained without carrying 
out the individual calculations of products and squares. Such methods as 
digiting, progressive digiting, and digiting without sorting have been discussed 
(1, 2, 3, 4). In general, one run through of the tabulator is required for each 
position of the multiplier. The partial totals thus obtained must then be 
added to secure the final totals. More recently, with the introduction of the 
electronic computing punch, it has become possible to obtain the summed 
products of variables up to a total of twenty digits on a trailer card (5). 
However, this method requires considerable time and is more subject to error 
than the methods mentioned above. 

The rolling totals method described below appears to have several distinct 
advantages over those methods previously described. The main advantages 
are: 

1. It saves considerable time and eliminates the need for a summary 
punch. Although the rolling totals method gives up counter space used in 
other methods for obtaining progressive totals for more variables, it still saves 
time by omitting three usual procedures. In this method there is no change of 
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wiring within each set of variables. The number of card cycles is greatly 
reduced because there are no intermediate totals printed and no summary cards 
punched. Finally, the process of summing the summary cards, with all that 
it entails, is eliminated. 

2. This method provides immediate independent visual checks on the 
accuracy of the sums of cross-products of each successive variable as it is 
run. Generally speaking, the fewer steps involved in a computational process, 
the less chance of error. After the second run there is an immediate visual 
check on the cross-products X,X, and X,X,. The only possible errors, either 
in wiring the variables from the lower brushes or in a mis-sort, are caught 
immediately. The method has proved to be so accurate that a straight card 
count or an accumulating card count to obtain a check on the sums for each 
variable is actually unnecessary and takes up counter space that could be 
used for another variable. 

3. Controlling is done by sorting so that wires need not be changed with 
each run through on the tabulator. 

The values for >) X7 and >> X;X; are obtained directly and the final 
totals are printed on one sheet as a symmetrical matrix, with the cells of the 
principal diagonal containing the sums of squares and the remaining cells 
the sums of cross-products. Table 1 shows the algebraic set-up for n variables. 

The values for >> X; are obtained by summing the detail cards on the 
tabulator without change of wiring. The basic principle for computing >> X7 
and >, X,X; is that of (a) sorting digit cards after detail cards, and (b) 
summing the detail cards on the tabulator by the “rolling totals” method. 
This procedure is repeated for each variable. 


Equipment 
To perform the operations described below, the only pieces of IBM 


equipment required are a sorter and a tabulator equipped with the Card 


TABLE 1 
Algebraic Illustration of Sums Obtained by the Rolling Totals Method 

















Variables Sorted on 
_- es a races a 

Sums [2X > X: rap. e —_ — 2X, J none 
Sums of [ = Xt _ X1X_e > X 1X 3 a >) Ainhs Xi 
Squares Zz XX; Das 2 XX3 —_- — 2D X2Xn X2 

and Z X3X1 Z X3X2 2 4 =. — z X3Xn X3 
Sums of | a os a —_- — a oo 
Cross- ; — — — -_-_ — — 
Products LZ xX.X, = XX: ZxXXs —_- — a a 
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Cycle Total Transfer Device, or “rolling” device for short.* Tabulators may 
be equipped with this device, replacing progressive digiting switches which 
will then be unnecessary. Moreover, since no summary punch or other equip- 
ment is required, this method is more economical than others. 


Procedure 


The sums of squares and cross-products are obtained by a single run of 
the cards through the tabulator for each variable. The cards are sorted 
together with a set of X-punched digit cards in descending order on the field 
of the multiplier, that is, the field to be squared. For a one-column multi- 
plier, a maximum of nine digit cards (9 to 1), for two columns, ninety-nine 
cards (99 to 01), and for three columns, 999 cards (999 to 001) will be required. 
When the method is used a great deal, it is more efficient to have three sets 
of digit cards. The one-column set has 9’s punched continuously in columns 
1 to 79, another has 8’s, another 7’s, and so on through 1’s. One of the 
two-column sets has 99’s punched continuously, another 98’s, another 97’s, 
and so on through 01’s in columns 1 to 78. The other two-column set starts 
at column 2 and goes through 79. All cards are X-punched in column 80. 

Now if there are no detail cards for a number, at least a digit card for 
that number will be present. All digit cards larger than the largest number 
in any variable are removed as these are unnecessary. The file of cards is now 
sorted with the digit cards following the detail cards. The last card of the 
file will be a digit card numbered 1 or 01. The cards are tabulated on an 
accounting machine which is equipped with the rolling device on counters 
2A, 2B, 4A, 4B, 6A, 6B, 8A, and 8B—the A-B counter group. The totals 
accumulated are rolled from the A-B counters into the C-D counters by the 
digit cards. When the last digit card passes through the tabulator, the total 
sums are printed. 


Method 


Figure | shows the plugging for obtaining the sums, sums of squares, and 
sums of cross-products for six variables X, to X, . The top counters A-B 
are paired with the bottom counters C-D. By making all the A’s and B’s 
common and all the C’s and D’s common, a maximum of forty counting units 
is available in each group. The sums are rolled from the A-B counter group 
into the C-D counter group on each digit card. All counters are usually 
equipped with the Card Cycle Total Transfer Device, although only the A-B 
counters need be for the method proposed here. 

The variables are plugged to the counter group A-B through the No-X 
side of Selectors E and Ff in order to prevent accumulating from the digit 

*This device can be attached to type 405 and type 416 Accounting Machines. It 
comes as standard equipment on the new type 407. We used a type 416 tabulator and a 
type 080 sorter. 


tSelectors E and F are not standard equipment on all machines. However, Selectors 
A, B, C, and D can be used. Selectors E and F were used because of convenience in wiring. 
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cards. The add and subtract impulses are also under control of a selector, 
in this case Selector 1. When the digit card passes the upper brushes, the 
X in column 80 sets up Selectors E and F so that when this card passes the 
lower brushes, these selectors are in the controlled position and accumulation 
from the digit card is eliminated. When the digit card passes the lower 
brushes, counter group A-B subtracts and the totals standing in it are trans- 
ferred to counter group C-D, which accumulates the totals from each digit 
card. After the transfer, counter group A-B will contain the same figures as 
before the transfer. If there are no detail cards but only a digit card present 
for a group, the totals transferred from the previous group will be transferred 
again for this group. After all the cards have passed through the machine, 
counter group C-D contains the sum of squares and sums of cross-products 
for the variable sorted. These totals are wired to print as final totals. 

Card count,is optional. It is wired in this illustration through Selector E 
as for the variables. Progressive accumulation was not wanted; therefore, 
counters 2A and 2B are wired directly to Type Bar Total Entry. Counter 
Total Control for card count is wired to Intermediate for printing on the same 
cycle as the final totals. If card count is wired the same as a variable and 
allowed to roll with the other totals, the result will be >> X; for the variable 
sorted. This provides an extra check on the accuracy of the work. 

Table 2 shows an actual run by the rolling totals method with appropriate 
headings added. The column headings indicate the columns in which the 


TABLE 2 


Sample of Sums, Sums of Squares and Sums of Cross-Products 
Obtained by the Rolling Totals Method 











Card 

Variables Count X Xe X3 X43 X35 X¢6 
Column 1 3 66 66 67 77 77 
Headings 1 1 56 78 90 12 34 
Sums 206 1027 2771 2628 2772 2760 2808 
Sums of 206 5903 14484 13638 14448 14367 14517 
Squares 206 14484 40775 38084 40106 39364 40158 

and 206 13638 38084 37092 38316 37762 38597 
Sums of 206 14448 40106 38316 41304 40088 40702 
Cross- 206 14367 39364 37762 40088 40704 40332 
Products 206 14517 40158 38597 40702 40332 42382 





variables have been punched, and they are printed by running through the 
usual heading cards one at a time, followed by an X-punched card. The sums 
of the six variables are obtained by running the unsorted detail cards, followed 
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by a single digit card, through the tabulator. The matrix of the sums of 
squares and sums of cross-products is obtained by the method explained above 
—with only a single run of the sorted cards for each variable. At no time is it 
necessary to change the wiring because the summing is controlled by the 
sorting. Since row indication cannot be printed automatically, care must be 
taken to sort each variable in order. 


Final Comments 


Since the maximum number of counters available is usually forty,* it 
will be possible to operate with ten variables if the sums of squares never 
exceed four digits, with eight variables for five digits, and so on. When all 
forty counters are required for the variables, card count can be omitted. If 
more than forty counters are needed, it will be necessary to do the runs in two 
or more parts. In this case, the partial matrices will be on two or more sheets 
which can be fitted together to make a complete symmetrical matrix. 

An operator dealing with a minimum of about 200 cards in each of two 
or more decks can keep a tabulator running continuously. Therefore, if one 
is operating with only one set of cards, it is often more economical to gang 
punch a second deck and sort on alternate variables. 


REFERENCES 


1. Castore, George F., and Dye, William §., III. A simplified punch card method of 
determining sums of squares and sums of products. Psychometrika, 1949, 14, 243-250. 

2. Deemer, Walter L. (Editor). Records, analysis, and test procedures. AAF Aviation 
Psychology Program Research Reports, No. 18. Washington: U. S. Government 
Printing Office, 1947, 405-442. 

3. Dwyer, Paul S., and Meacham, Allan D. The preparation of correlation tables on a 
tabulator equipped with digit selection. J. Amer. statist. Ass., 1937, 32, 654-662. 

4. Ellis, Max E., and Riopelle, Arthur J. An efficient punched-card method of computing 

> X, = X*, and ZY XY, and higher moments. Psychometrika, 1948, 13, 79-85. 

International Business Machines Corporation. Calculating punch, type 602A, prin- 

ciples of operation. New York: IBM, 1948. 


oa 


*Actually fifty-six with the new type 407. 


Manuscript received 10/24/51 


Revised manuscript received 12/31/51 








PSYCHOMETRIKA—VOL. 17, No. 3 
SEPTEMBER, 1952 


AN EXTENDED TABLE OF CHI-SQUARE FOR TWO DEGREES 
OF FREEDOM, FOR USE IN COMBINING PROBABILITIES 
FROM INDEPENDENT SAMPLES 


Morpecal H. Gorpon, Epwarp H. LovELAND, AND EpwarpD E. CURETON 


VA HOSPITAL, CHILLICOTHE, OHIO; GEORGIA INSTITUTE OF TECHNOLOGY ; 
AND UNIVERSITY OF TENNESSEE 


A table of values of Chi-square for two degrees of freedom corresponding 
to values of P from .001 to .999 is presented, together with a description and 
an example of its use in combining probabilities from two or more independent 
samples to obtain an aggregate probability. 


Duplicate studies, and studies bearing on the same essential problem 
but based on samples from non-identical populations, often lead to tests of 
significance whose associated probabilities are quite unequal. If each sample 
is small, moreover, the null hypothesis may not be rejected with any great 
certainty on the basis of any one study. But if the results of the several 
studies are all or nearly all in the same direction, there is additional evidence 
against the null hypothesis. In such cases we may wish to base a new test 
of significance on the combined data from all the studies. 

Fisher (2) has described a method for combining the probabilities yielded 
by two or more studies, by making use of the fact that the distribution of the 
sum of several values of Chi-square is itself a Chi-square distribution, and 
that the value of Chi-square for two degrees of freedom is —2 times the 
natural logarithm of the probability. To use this method it is necessary to 
find from each original study the exact probability corresponding to the 
statistic involved in the test of significance. 

Since many of the commonly used tables do not give directly the natural 
logarithms of the numbers between 0.000 and 1.000, the preparation of an 
extended direct table of values of Chi-square appeared desirable. The accom- 
panying table gives four-decimal values of Chi-square, for two degrees of 
freedom, corresponding to values of P from .001 to .999. These values were 
found by doubling the entries in a five-decimal table of natural logarithms (3), 
changing the signs, and rounding off to four decimals. For values of P greater 
than .05 and accurate to four or five decimals, linear interpolation will be 
accurate enough for most practical purposes. For values of P less than .05 
and accurate to more than three decimals, the corresponding values of Chi- 
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TABLE 1 


Values of Chi-Square for Two Degrees of Freedom 
Corresponding to Values of P from .001 to .999. 











PO 1 2 3 4 5 6 7 8 9 

00 13.8155 12.4292 11.6183 11.0429 10.5966 10.2320 9.9237 9.6566 9.4211 
01 9.2103 9.0197 8.8457 8.6856 8.5374 8.3994 8.2703 8.1491 8.0348 7.9266 
02 7.8240 7.7265 7.6334 7.5445 7.4594 7.3778 7.2993 7.2238 7.1511 7.0809 
03 7.0131 6.9475 6.8840 6.8225 6.7628 6.7048 6.6485 6.5937 6.5403 6.4884 
04 6.4378 6.3884 6.3402 6.2931 6.2471 6.2022 6.1582 6.1152 6.0731 6.0319 
05 5.9915 5.9519 5.9130 5.8749 5.8375 5.8008 5.7648 5.7294 5.6946 5.6604 
06 5.6268 5.5938 5.5612 5.5292 5.4977 5.4667 5.4362 5.4061 5.3765 5.3473 
07 5.3185 5.2902 5.2622 5.2346 5.2074 5.1805 5.1540 5.1279 5.1021 5.0766 
08 5.0515 5.0266 5.0021 4.9778 4.9539 4.9302 4.9068 4.8837 4.8608 4.8382 
09 4.8159 4.7938 4.7719 4.7503 4.7289 4.7078 4.6868 4.6661 4.6456 4.6253 
10 4.6052 4.5853 4.5656 4.5461 4.5267 4.5076 4.4886 4.4699 4.4512 4.4328 
11 4.4145 4.3965 4.3785 4.3607 4.3431 4.3256 4.3083 4.2912 4.2741 4.2578 
12 4.2405 4.2239 4.2075 4.1911 4.1749 4.1589 4.1429 4.1271 4.1115 4.0959 
13 4.0804 4.0651 4.0499 4.0348 4.0198 4.0050 3.9902 3.9755 3.9610 3.9466 
.14 3.9322 3.9180 3.9039 3.8898 3.8759 3.8620 3.8483 3.8346 3.8211 3.8076 
15 3.7942 3.7810 3.7677 3.7546 3.7416 3.7287 3.7158 3.7030 3.6903 3.6777 
16 3.6652 3.6527 3.6403 3.6280 3.6158 3.6036 3.5915 3.5795 3.5676 3.5557 
17 3.5439 3.5322 3.5205 3.5089 3.4974 3.4859 3.4745 3.4632 3.4519 3.4407 
18 3.4296 3.4185 3.4075 3.3965 3.3856 3.3748 3.3640 3.3533 3.3426 3.3320 
19 3.3215 3.3110 3.3005 3.2901 3.2798 3.2695 3.2593 3.2491 3.2390 3.2289 
20 3.2189 3.2089 3.1990 3.1891 3.1793 3.1695 3.1598 3.1501 3.1404 3.1308 
21 3.1213 3.1118 3.1023 3.0929 3.0836 3.0742 3.0650 3.0557 3.0465 3.0374 
.22 3.0283 3.0192 3.0102 3.0012 2.9922 2.9833 2.9744 2.9656 2.9568 2.9481 
.23 2.9394 2.9307 2.9220 2.9134 2.9049 2.8963 2.8878 2.8794 2.8710 2.8626 
.24 2.8542 2.8459 2.8376 2.8294 2.8212 2.8130 2.8048 2.7967 2.7887 2.7806 
25 2.7726 2.7646 2.7567 2.7487 2.7408 2.7330 2.7252 2.7174 2.7096 2.7019 
26 2.6941 2.6865 2.6788 2.6712 2.6636 2.6561 2.6485 2.6410 2.6335 2.6261 
27 2.6187 2.6113 2.6039 2.5966 2.5893 2.5820 2.5747 2.5675 2.5603 2.5531 
28 2.5459 2.5388 2.5317 2.5246 2.5176 2.5105 2.5035 2.4965 2.4896 2.4827 
.29 24757 2.4689 2.4620 2.4552 2.4484 2.4416 2.4348 2.4280 2.4213 2.4146 
30 2.4079 2.4013 2.3947 2.3880 2.3815 2.3749 2.3683 2.3618 2.3553 2.3488 
31 2.3424 2.3359 2.3295 2.3231 2.3167 2.3104 2.3040 2.2977 2.2914 2.2851 
32 2.2789 2.2726 2.2664 2.2602 2.2540 2.2479 2.2417 2.2356 2.2295 2.2234 
33 2.2173 2.2113 2.2052 2.1992 2.1932 2.1872 2.1813 2.1753 2.1694 2.1635 
34 2.1576 2.1517 2.1459 2.1400 2.1342 2.1284 2.1226 2.1169 2.1111 2.1054 
35 2.0996 2.0939 2.0882 2.0826 2.0769 2.0713 2.0656 2.0600 2.0544 2.0489 
36 2.0433 2.0378 2.0322 2.0267 2.0212 2.0157 2.0102 2.0048 1.9993 1.9939 
37 1.9885 1.9831 1.9777 1.9724 1.9670 1.9617 1.9563 1.9510 1.9457 1.9404 
38 1.9352 1.9299 1.9247 1.9194 1.9142 1.9090 1.9038 1.8987 1.8935 1.8884 
.39 1.8832 1.8781 1.8730 1.8679 1.8628 1.8577 1.8527 1.8476 1.8426 1.8376 
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TABLE 1—Continued 














40 1.8326 1.8276 1.8226 1.8176 1.8127 1.8077 1.8028 1.7979 1.7930 1.7881 
41 1.7882 1.7783 1.7735 1.7686 1.76388 1.7590 1.7541 1.7493 1.7445 1.7398 
42 1.73850 1.7302 1.7255 1.7208 1.7160 1.7113 1.7066 1.7019 1.6973 1.6926 
43 1.6879 1.6833 1.6787 1.6740 1.6694 1.6648 1.6602 1.6556 1.6511 1.6465 
44 1.6420 1.6374 1.6329 1.6284 1.6239 1.6194 1.6149 1.6104 1.6059 1.6015 


45 1.5970 1.5926 1.5881 1.5837 1.5793 1.5749 1.5705 1.5661 1.5618 1.5574 
46 1.5531 1.5487 1.5444 1.5401 1.5857 1.53814 1.5271 1.5229 1.5186 1.5143 
47 «1.5100 1.5058 1.5016 1.4973 1.4931 1.4889 1.4847 1.4805 1.4763 1.4721 
48 1.4679 1.4638 1.4596 1.4555 1.4513 1.4472 1.4431 1.4390 1.4349 1.4308 
49 1.4267 1.4226 1.4186 1.4145 1.4104 1.4064 1.4024 1.3983 1.3943 1.3903 


50 1.8863 1.3823 1.3783 1.38743 1.3704 1.3664 1.3624 1.3585 1.3545 1.3506 
51 1.38467 1.3428 1.3389 1.3350 1.3311 1.38272 1.3233 1.3194 1.3156 1.3117 
52 1.3079 1.3040 1.3002 1.2963 1.2925 1.2887 1.2849 1.2811 1.2773 1.2735 
.53 1.2698 1.2660 1.2622 1.2585 1.2547 1.2510 1.2472 1.2485 1.2398 1.2361 
54 1.2324 1.2287 1.2250 1.2213 1.2176 1.2139 1.2103 1.2066 1.2030 1.1993 


55 1.1957 1.1920 1.1884 1.1848 1.1812 1.1776 1.1740 1.1704 1.1668 1.1632 
56 1.1596 1.1561 1.1525 1.1490 1.1454 1.1419 1.1383 1.1848 1.1313 1.1277 
57 1.1242 1.1207 1.1172 1.1137 1.1103 1.1068 1.1033 1.0998 1.0964 1.0929 
58 1.0895 1.0860 1.0826 1.0791 1.0757 1.0723 1.0689 1.0655 1.0621 1.0587 
59 1.0553 1.0519 1.0485 1.0451 1.0418 1.0384 1.0350 1.0317 1.0283 1.0250 


60 1.0217 1.0183 1.0150 1.0117 1.0084 1.0051 1.0018 .9985  .9952 .9919 
61 9886 .9853 .9820 .9788 .9755 .9723 .9690 .9658  .9625 .9593 
62 9561 .9528 .9496 .9464 .9432 .9400 .9368  .9336 .9304 .9272 
.63 9241 .9209 .9177 .9146 .9114 .9083 .9051 .9020 .8988  .8957 
64 8926 .8895 .8863 .8832 .8801  .877 8739 .8708 .8677 .8646 


65 8616 .8585 .8554 .8524 .8493 .8462 .8432 .8401 8371 .8341 
-66 .8310 .8280 .8250 .8220 .8189 .8159 .8129 .8099 .8069 .8039 
67 8010 .7980 .7950 .7920 .7891 .7861 .7831 .7802 .7772 = .7743 


68 77138 = 7684 — «67655 «67625 «£7596 = 67567 = 7588S £7508 = 7479 ~—«.7450 
69 7421 = 7392) 7363) 738385) 73060 .7277)S £7248) £7219) «.7191 ~— .7162 
70 71383-7105. «.7076 = «.7048)~— 67020 «6991 = 6963 = «6934 = .6906 ~—.6878 
i! .6850 .6822 .6794 .6765 .6737 .6709 .6682 .6654 .6626 .6598 
2 6570 = .6542) 66515 66487) = 6459 = 64826404 = 6377 = 6349 = 6322 
.73 .6294 .6267 = .6239 .6212 6185 .6158 6131 .6103 .6076 .6049 
74 6022 .5995 .5968 .5941 .5914 .5887 .5861 5834 .5807 .5780 


75 5754 .5727 .5700 .5674 3 3=.5647 = .5621) = 559405568) = 55415515 
76 5489 .5462 5486 «=6.5410) Ss 53840 65358) = 5831) = 653805) .5279 ~~ 5253 
Sith 5227 =.5201_)—=—s 5175S «5150S 5124. 5098 )3=—s- 5072.) .5046=— 5021 = 4995 
78 4969 .4944 .4918 .4892 4867 4841 4816 4791 .4765 .4740 
9 4714 =.4689~—s 46644639 46134588) 456345884513 .4488 
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TABLE 1—Continued 

















.80 4463 4488 .4413 .4388 .4363 .43838 4313 .4289 .4264 .4239 
81 4214 4190 .4165 .4140 .4116 .4091 .4067 .4042 .4018 .3993 
82 .3969 .3945 .3920 .3896 .3872 .3847 .3823 .3799 .38775 .3751 
83 3727 ~=.3703—S 3678 ~— 3654 = «36380 )=— «3606 = 3583) .8559 = «85358511 
84 3487 .3463 = 3440S 3416 = 3392) 3368) = 3345 = 332128297 ~=—s 8274 


85 3250 .3227 =©.3203—s «.3180)—s 3156 = .3133)—ss«3110 = .3086)3=—s 3063 = .3040 
86 3016 .2993 .2970 .2947 .2924 .2901 .2877 .2854 .2831 .2808 
87 2785 .2762 =.2739—Ss 2716 ~—S .2693)=— 2671 = «.2648 3S £2625 = £2602 ~=— 2579 
88 2557 =.2534.—S 2511) 2489S 2466 )= 24438) Ss £2421) = 2898 )3=— 2376 ~— 23538 
89 .2331 .2308 8.2286 )=—«. .2263—S_—( 2241) 2219) Ss 2196 )=—s 2174 —Ss 2152S 2129 


90 2107 =.2085 = .2063) =.2041 +3=.2019 Ss 1996 )=—s- «1974 §=.1952)—s .1980 ~— 1908 
91 1886 .1864 .1842 .1820 .1798 .1777 .1755 .1783 .1711 = .1689 
92 .1668 .1646 .1624 .1603 .1581 .1559 .15388 .1516 .1494 .1473 
.93 1451 .1480 = .1408 .1387 = .1366 =6.1844_ —s .1823'—Ss «1301 =—s «.1280~——.1259 
94 1288 =.1216 = 1195S «1174S 1153s 1181~—S 1110S 1089 —s 1068 ~— 1047 


95 1026 .1005 .0984 .0963 .0942 .0921 .0900 .0879 .0858 .0837 
96 0816 .0796 0775 .0754 .0733 .0713 .0692 .0671 .0650 .0630 
97 .0609 .0589 .0568 .0547 .0527 .0506 .0486 .0465 .0445 .0424 
98 0404 .0384 .0363 .0343 .0323 .0302 .0282 .0262 .0241 .0221 
.99 0201 .0181 .0161 .0140 .0120 .0100 .0080 .0060 .0040 .0020 





square for two degrees of freedom may be found by multiplying their common 
logarithms by —4.60517. 

To find the combined probability from n duplicate or parallel studies, 
find the value of Chi-square (for two degrees of freedom) corresponding to 
the one-tail P yielded by each study. If some of the studies yield results of 
opposite sign from the majority, the values of Chi-square to be used are those 
corresponding to (1 — P) rather than P. Add these Chi-square values. The 
sum may be treated as a Chi-square based on 2n degrees of freedom, the P 
corresponding to this Chi-square being the probability for the combined 
significance test. 

Example: McNemar and Terman (5) report data from studies by Thorn- 
dike based on the administration of a paper-and-pencil intelligence test (the 
CAVD) to similarly selected samples of boys and girls from 13 to 17 years 
of age. Each of thirteen age groups in various cities was treated as a separate 
sample. The critical ratios of the differences between pairs of standard devia- 
tions were computed by large-sample methods. The thirteen pairs of N’s, 
critical ratios, one-sided probabilities, and Chi-square values for two degrees 
of freedom are as follows: 
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Sample N (boys) N (girls) CR P Chi-square 
1 252 279 1.00 .159 3.6777 
2 555 637 1.28 .100 4.6502 
3 439 490 2.04 .021 7.7265 
+ 170 155 29 .386 1.8987 
5 220 274 24 .405 1.8077 
6 377 494 1.61 .054 5.8375 
z 373 405 1.22 mL 4.3965 
8 153 189 .33 371 1.9831 
9 180 198 .50 .309 2.3488 

10 615 716 .66 255 2.7330 
11 775 1054 ila 74 .121 4.2239 
12 728 865 — .38 648 .8677 
13 385 436 .18 .429 1.6926 

43.7989 


At 26 degrees of freedom, the value of the combined Chi-square is significant 
at the .02 level but not at the .01 level. This is a one-sided test, however, 
while the hypothesis (no difference in variability as between boys and girls) 
calls for a two-sided test. The tabled significance values must therefore be 
doubled, and we conclude that the hypothesis is to be rejected at the .04 
level but not at the .02 level. The positive CR’s in this example refer to 
pairs of samples in which the variability of boys exceeds that of girls. Earlier 
studies would have led the authors to state this as the alternative to the null 
hypothesis. In that case the one-sided test would apply, and the null hy- 
pothesis would be rejected at the .02 level but not at the .01 level.* 

It has been pointed out (4, 6) that when the P’s to be combined come 
from four-fold tables or binomial distributions with some one or more fre- 
quencies small, this method of combining probabilities is not accurate. In 
such cases P can take only discrete values, and the neighboring values are 
not very close together. Fisher’s method of combining probabilities applies 
strictly only when the original tests of significance are of such nature that all 
values of P are equally probable a priorit. When values of P do come from 
discrete distributions, however, the use of Fisher’s method is “‘conservative’’; 
i.e., the null hypothesis will be rejected less often by this test than it will by 
a more exact test. 

A convenient list of statistical tables, including most of the original 
tables which yield exact probability values from significance tests, has been 
given recently by Bancroft (1). 


*One reviewer of this paper suggests that in cases such as the one in this example, 
where the significance test is based on a critical ratio (which already assumes a normal 
sampling distribution), we may use the t-test of the hypothesis that the mean CR is zero, 
taking 1/n times the variance of the CR’s in the n samples as an estimate of the sampling 
variance of these means. The weighted mean CR in the present example is .822, the 
standard error of this mean is .202, and t = 4.07, which for 12 degrees of freedom is sig- 
nificant (in a two-sided test) at the .01 level but not at the .001 level. In this instance 
the é-test is more sensitive than is the Chi-square test based on the combined distribution. 
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ORTHOGONAL AND OBLIQUE SIMPLE STRUCTURES 


W. A. GiBson 


UNIVERSITY OF ILLINOIS 


A quick approximation of the best-fitting orthogonal simple structure 
from the known oblique simple structure is here developed. 


Introduction 


For some time the proponents of oblique simple structure in multiple- 
factor analysis have more or less informally used the cosines of the angles 
between the oblique reference vectors as ready estimates of the correlations 
between the corresponding primary factors (cf. 3, p. 212). This is because 
of the fortunate circumstance that the cells in the matrix showing the cosines 
of the angles between pairs of reference vectors (C = A’A, where A is the 
oblique transformation matrix) have the same pattern of magnitudes, but with 
opposite signs in the side entries, as do the cells in the matrix of correlations 
between the primaries (R,, = 77’, where each row of T shows the projections 
of the associated primary on the initial orthogonal reference frame). An 
example of this is Thurstone’s famous box problem (3, 131-146). The 
matrices C and R,, for that three-dimensional example are shown in Table 1. 




















TABLE 1* 
Box Problem C and Ry, 
C= A’‘A Roe = TT’ 
AS My As tT; Ty T, 
Az 1.000 —.218 —.055 Ts 1.000 . 229 105 
Ay — .213 1.000 —.206 ly . 229 1.000 224 
A, —.055 —.206 1.000 T, .105 224 1.000 








*Reproduced from p. 136 of (8) by permission of the University of Chicago Press. 


The purpose of the present paper is to show that this condition can be 
utilized to provide, for those who are addicted to orthogonal solutions (cf. 2, 
42, and 4, 241), a close approximation to the best-fitting orthogonal simple 
structure when the oblique simple structure position is known. In Figure 1 
we have pictured the right spherical triangle corresponding to the best-fitting 
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orthogonal simple structure for a configuration of test vectors in which the 
termini of the primaries of the oblique solution are the three points within 
the triangle and the termini of the reference vectors of the oblique solution 
are the three points outside of the triangle. The appearance of this drawing 


Ay 
° 


Oy 


Oo; 





AS 
FIGurE 1 


suggests that each reference vector of the orthogonal solution is closely 
approximated by the simple average of the like-numbered primary and 
reference vectors of the oblique solution. Thus, for example, the unit vector 
which passes through the centroid of the vectors 7, and A, ought to be nearly 
identical with the orthogonal reference vector O, . 


Procedure 


To determine the extent to which such a set of centroids will constitute 
an orthogonal reference frame, we first form the matrix A + 7’ and normalize 
its columns. The columns of the resulting matrix will give the direction 
cosines of the centroids of pairs of corresponding primary and reference 
vectors of the oblique solution. We may designate this matrix as 


Q= (A+ T)K, (1) 


where K is the diagonal matrix of normalizing constants. In Table 2 we 
have the matrices A and 7” for the box problem (3, 136). Also shown in 
Table 2 are the matrices A + 7’, K, and Q, along with the necessary computa- 
tional operations and summational checks. We observe that the non-vanish- 
ing entries in K are very nearly of the same size, which is a trifle greater than 
.5. This is because each primary and its corresponding reference vector in 
the box problem are fairly close together. As associated primary and reference 
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TABLE 2* 
Box Problem A, T’, A + T’, K, and Q 
A a 
Az Ay we Tz Ty Ts 
I .483 .466 .479 I .661 .713 .654 
II — .834 . 254 .560 II —.736 .199 .545 
lil . 267 — .847 675 Ill 144 —.671 .524 
= — .084 — .127 1.714 2 .069 . 241 1.723 
A+T' Q=(A+T)K 
x y z Qz Qu Q. 
I 1.144 1.179 1.133 [ .576 .597 .571 
II -—1.570 .453 1.105 II —.791 . 229 .557 
III -411 —1.518 1.199 III .207 —.769 .604 
Ch. — .015 .114 3.437 Ch. — .008 .058 1.731 
z — .015 .114 3.437 = —.008 .057 1.732 
zi? 3.9426 3.8996 3.9423 
V ZI? 1.9856 1.9747 1.9855 
K . 5036 .5064 .5037 





*The matrices A and 7’ reproduced from p. 136 of (8) by permission of the University of Chicago Press. 


vectors approach éach other, the corresponding diagonal entry in K approaches 
the value .5. 


Empirical and Theoretical Verification 


The matrix product Q’Q will give the cosines of the angles between the 
unit reference vectors defined by the columns of Q, and hence will indicate 
how nearly orthogonal are these reference vectors. The product Q’Q for the 
box problem is shown in Table 3. We find that the side entries in Table 3 
very nearly vanish, from which we conclude that Q is quite close to being 
an orthogonal transformation matrix. 


TABLE 3 
Box Problem Q’Q 











Q: Qy Q: 
Qz 1.000 .004 .013 
Qy .004 1.000 .004 


Q: .013 .004 1.001 
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Let us now look into the reason for this by carrying out the multiplication 
Q’Q symbolically. For this purpose we shall need the relationship 


TA = D, (2) 


which is given by Thurstone (3, 137). Here the diagonal matrix D shows 
the cosines of the angles between primary vectors and reference vectors. 
D is diagonal because each primary is orthogonal to all but one of the reference 
vectors and each reference vector is orthogonal to all but one of the primaries. 
Now we may carry out the multiplication Q’Q as follows: 


QQ = K(A+7T)(A + T)K 

K(A’ + T)(A + 7’)K 

K(A’A + A’'T’ + TA + TT")K (3) 
K(C + D'+ D+R,,)K 

K(C + 2D + R,,)K. 


The quantity within parentheses in the last step of equation (3) is the sum 
of four symmetric matrices, two of which are diagonal, while the other two, 
as we have seen, have side entries which are nearly equal but are opposite in 


TABLE 4 
Box Problem Vg 











Q: Qy Q. 

1 .990 .119 .050 

2 .139 .979 .118 

3 .062 . 135 .981 

4 .574 .812 .112 

5 442 . 149 . 884 

6 . 102 .501 . 858 

7 795 .614 .075 

8 . 892 . 163 .440 

9 .105 . 839 543 

10 . 703 719 .099 
11 ./19 . 162 . 689 
12 101 705 714 
13 .980 . 148 .043 
14 .090 . 960 .173 
15 .077 .092 . 956 
16 .370 .491 INE 
7 . 128 . 586 . 368 
18 . 964 .081 .056 
19 181 .945 .071 


20 .060 . 144 .972 
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sign. Hence the side entries of this matrix sum are quite small, and the pre- 
and post-multiplication of this sum by K makes the resulting side entries 
even smaller. Since the diagonal entries in K are all just a little greater than 
1/2, we may reasonably summarize the effect of K by saying that it reduces 
all the entries in C + 2D + R,, to a little over 1/4 of their original size. 
Thus any side entry in Q’Q will be only a trifle larger than 1/4 of the algebraic 
sum of the corresponding entries in C and R,, . 

From the last step of equation (3) we see that the general diagonal entry 
in K is actually defined as the reciprocal of the square root of the corresponding 
diagonal value in the sum C + 2D + R,, .. That is, 

1 1 2 
he W14+3264+1 2Vi+d,° @) 
Thus when d, is reasonably large, k, is quite close to 1/2. We have already 
seen this to be true in the box problem. 

Applying the transformation matrix Q to the initial orthogonal factor 

matrix fy by means of the equation 


F.Q = Vo, (5) 





we obtain, in Vg , the projections of the test vectors on the nearly orthogonal 
axes defined by the columns of Q. The matrix Vg for the box problem is 
shown in Table 4. Figures 2, 3, and 4 are the plots of pairs of columns of 
Table 4. That this solution closely approximates the best-fitting orthogonal 
simple structure is evident. 


Conclusions 


This approximation procedure will work best with configurations whose 
oblique solutions are not too strongly oblique and are reasonably symmetric 
with respect to the angles between primaries or reference vectors. In any 
event, interested persons can find out in a rough way beforehand how good an 
approximation they will get by comparing the side entries in C and R,,. A 
more precise check on the orthogonality of Q would be its pre- and post- 
multiplication by its transpose, and that orthogonality could of course be 
improved by judicious successive adjustments of the entries in Q. Additional 
small rotations of the resulting Vg could be made as desired. 

After this paper was submitted for publication there was called to my 
attention an article by Green (1) which presents three different least-squares 
solutions to this same problem. The procedure described here may be thought 
of as a rapid approximation to either of Green’s two simpler solutions, in 
which the best orthogonal fit of the reference vectors or of the primary axes 
is sought, rather than a least-squares minimization of differences in factor 
loadings as in his first solution. 
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Summary 


We have suggested that the transformation matrix for an orthogonal 
simple structure solution may often be closely approximated simply by nor- 
malizing the columns of the sum of the matrices A and 7” of the oblique simple 
structure solution. 
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COMPANION NOMOGRAPHS FOR TESTING THE 
SIGNIFICANCE OF THE DIFFERENCE BETWEEN 
UNCORRELATED PERCENTAGES* 


VALENTINE APPEL 


RICHARDSON, BELLOWS, HENRY, & CO. 


The purpose of this article is to present a set of nomographs for testing 
the significance of the difference between uncorrelated percentages, and for 
determining necessary sample sizes when planning studies which will involve 
the comparison of percentages. Examples of these two uses are given as well 
as a description of the formula upon which the nomographs are based. 


Introduction 


Many short-cut methods of item analysis have been developed in recent 
years which test the significance of the difference between the proportions 
of cases in two independent samples responding to a test item. But since 
most of these techniques have been developed for specific purposes, they tend 
to suffer limitations in terms of the amount of computation necessary, the 
precision of the results, or the special assumptions involved particularly 
concerning sample sizes. 

A set of companion nomographs is reproduced here which permits an 
evaluation of the significance of the difference between uncorrelated per- 
centages while avoiding most of the above limitations. 


Testing the significance of the difference between uncorrelated percentages 


1. Compute the percentage in each of the two samples being compared 
which respond in a particular way to the item being analyzed. Compute the 
difference between these two percentages (P,; — P,). Then compute the 
percentage responding in this way (P) for all available samples combined. 
In item analysis this might include the middle group as well as the two tail 
groups.t 

2. Enter Nomograph I, connecting the values of NV, and N, , the sizes 
of the two samples, with a straightedge. Read the value of N{ from the center 
scale. 


*The writer wishes to express his thanks and appreciation to Dr. Harold A. Edgerton 
for his generous advice and assistance in the preparation of this paper, and to Mr. Herman 
Greenblatt for his painstaking work in preparing these nomographs for the printer. 

tActually, for many practical purposes, a sufficiently stable estimate of P can be 
obtained when the middle group is omitted. 

tN, as employed here, is defined in formula (2) of the section dealing with the deriva- 
tion of the nomographs. 
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3. Enter Nomograph II, connecting the values of N and P withastraight- 
edge. Read the value of op,_p, on the center scale. 

4. Connect the value of op,_p, and the value of P, — P, with a straight- 
edge. Read the value of ¢ on the center scale. 

Consider the example presented in Table 1. The problem here is to 
determine whether alternative x of item y is able to discriminate the high 


TABLE 1 
Per Cent of Cases Marking Alternative x of Item y 




















High Middle Low All 
Criterion Criterion Criterion Groups 
Group Group Group Combined 
Alternative x 59% (P:) 49% 42% (P2) 49% (P) 
All other alternatives 41% 51% 58% 51% 
Sample size 135 (N:) 226 163 (N2) 524 





criterion group from the low criterion group. This can be done by testing 
the significance of the difference between the percentages marking alternative 
zx in the high and low criterion groups: 

1. The values of P,; and P, have already been computed and are pre- 
sented in Table 1. The value of P; — P, = 59% — 42% = 17%. The value 
of P (49%) is shown in the last column of Table 1. 

2. Entering Nomograph I, connecting the values of NV, and N, , we find 
N to be 74. 

3. Entering Nomograph II, connecting the values of N and P, we find 
Op,-p, tO be 5.8. 

4, Connecting the value of op,_p, and the value of P, — P, we find 
t to be 2.9. 

This ¢, when interpreted as a unit normal deviate, is significant beyond 
the 1% level of confidence. We may conclude, therefore, that alternative x 
of item y is statistically discriminative of these high and low criterion groups. 


Determining necessary sample sizes when planning studies 


Although these nomographs have been constructed primarily for the 
purpose of testing the significance of the difference between percentages, they 
are also useful in determining necessary sample sizes when planning studies 
which will involve the comparison of percentages.* 

It frequently happens that investigators decide, before the data collec- 
tion is begun, that some particular percentage difference shall be considered 
meaningful. The question then arises as to what sample sizes will be necessary 


*For a previous treatment of this problem see: Swineford, F. Graphical and tabular 
aids for determining sample size when planning experiments which involve comparisons 
of percentages. Psychometrika, 1946, 11, 43-49. 
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for this to be true. This problem can be solved simply by referring to Nomo- 
graphs I and II: 

1. Decide upon the percentage difference which will be considered mean- 
ingful. Locate this value on the P,; — P, scale in Nomograph II. 

2. Decide upon the level of significance required and locate the corre- 
sponding ¢ value. 

3. Connect the values of P, — P, and ¢ with a straightedge. Read the 
value of op,_p, . 

4. Connect the value of op,_p, with the value of P.* Read the value of N. 

5. Entering Nomograph I with the value of N, the combinations of 
permissible sample sizes can be read from the N, and N,j scales by rotating a 
straightedge around the value of N. If it is decided that N, should equal N, , 
then each sample size will equal 2N. 

It will be noted that unless the ratio of the sample sizes (N,/N-.) is known 
beforehand, as in the case of stratified random sampling, or unless one of the 
sample sizes is already specified, as would be the case if only a limited sample 
of a particular type were available, no unique solution for determining the 
necessary sample sizes is possible. 

Let us assume that we are about to undertake a study, and that we are 
willing to accept as meaningful a 10% difference significant at the 5% level. 
Let us further assume that we have no knowledge of how the combined groups 
will respond to the items to be employed, and that the sample sizes for each 
group will be equal: 

1. Locate the 10% point on the P, — P, scale of Nomograph II. 

2. Connect this point and the ¢ value of 1.96 (which is the normal deviate 
corresponding to the 5% level of confidence) with a straightedge. Reading 
the value of op,_p, we find it to be 5.2. 

3. Connect the 50% point on the P scale with the obtained value of 
op,-p,- Reading the value of N, we find it to be 95. 

4, Since we have assumed the two sample sizes to be equal, we multiply 
N by 2 in order to determine the two sample sizes. N, and N, are, therefore, 
each equal to 190. In other words each sample must contain at least 190 
cases before we can be sure that a 10% difference will be significant at the 5% 
level. 

Derivation 


The formula employed here for the standard error of the difference 
between two uncorrelated percentages ist 


or,-7, = Po? +3), (1) 


*If the value of P (the percentage of cases responding in a given way in the combined 
sample) is unknown, as it usually is, let P equal 50%. This will maximize the estimated 
necessary sample size, thus assuring that percentage differences of the predesignated magni- 
tude will be significant at the required level. 

tMcNemar, Q. Psychological Statistics. New York: John Wiley & Co., 1949. 
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when P refers to the percentage of cases in all the combined available samples 
responding in a particular way to the item being analyzed, Q equals 100% — P, 
and N, and N, refer to the sizes of the two samples being compared. 

Since the technique presented here seeks to test the null hypothesis that 
all the sample percentages can be considered random samples from the same 
universe, the most stable estimate of that universe value (P) is based upon all 
the samples combined. Thus in the example presented in Table 1, although 
the percentage difference being tested was that between the high and the low 
criterion groups, the best estimate of the universe percentage was based upon 
the combined sample of high, middle, and low criterion groups. 

By definition, let 


] 1 1 
WV, + N,N’ (2) 


then formula (1) may be rewritten as follows: 


seinen (3) 


The formula for testing the significance of the difference between two 
uncorrelated percentages therefore becomes: 
P, —P. 
¢ ae Fee, (4) 
PQ 
N 











The functions of Nomographs I and II* are to solve formulas (2) and (4), 
respectively. 
*The nomographs reproduced here are too small for efficient use. Single copies of 


useable size may be obtained free of charge by writing to the author at Richardson, 
Bellows, Henry & Co., 1 West 57th Street, New York 19, N. Y. 


Manuscript received 1/3/52 


Revised manuscript received 2/20/52 
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ON THE ECONOMIES OF A PRE-SCREENING TECHNIQUE 
FOR APTITUDE TEST BATTERIES* 


A. G. ArBous AND H. S. SIcHEL 


NATIONAL INSTITUTE FOR PERSONNEL RESEARCH 
SOUTH AFRICAN COUNCIL FOR SCIENTIFIC AND INDUSTRIAL RESEARCH 


Test battery selection is justified on the grounds that it is more efficient 
than the conventional methods. More precise definitions of efficiency have 
shown that the additional cost of test administration can often outweigh the 
gains of selecting a more competent group of persons. Final evaluation of the 
profit and loss account can be made only when no further economies can be 
effected in the selection procedures themselves. A psychometric pre-screening 
technique which will reduce this cost considerably is developed here. Its 
psychometric structure is mathematically defined, and its advantages illus- 
trated in terms of testing economy (E) and losses (L) of good material for 
different selection ratios (S. R.). It is shown that, under many circumstances, 
the testing economy can be considerable, and that in all practical cases 
“Losses” will be < 1.5% for all values of p > .8 (between the pre-screening 
rp and the final battery). The technique in practice is illustrated from a set 
of data. 


I. Introduction 


Previous investigators have devised several techniques for demonstrating 
the gains of aptitude test procedures over the more conventional methods of 
selection (1 through 14). One of these authors, Brogden (11, 14) has empha- 
sized that the cost of test administration per hired employee must be taken 
into consideration before the profit and loss account of testing programmes 
can be drawn up. He also indicates in these terms how testing can be an 
unprofitable undertaking in many circumstances. 

Even if one concedes that evaluation is to be made solely in terms of a 
“dollar criterion” and immediate financial gains, Brogden’s comments would 
apply only when the cost of testing cannot be further reduced by some eco- 
nomic pre-screening device—it being remembered that such a device should 
not result in other losses of potential successes which offset gains. 

A psychometric pre-screening device which has these advantages has 
already been devised by Arbous (8) to meet the need for further economy 
in a testing program where administration costs were high. In this par- 
ticular case 56% of the applicant group could be screened out in the first 
instance by means of a simple pencil-and-paper test, thereby reducing the 





*The authors wish to thank the staff members of the Statistical and Industrial Research 
Teams of the National Institute for Personnel Research for their assistance in the computa- 
tional work, preparation of diagrams, and editing of this paper. 
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application of expensive selection procedures to only 44%. This was achieved 
at the loss of only 1.4°% of those who would have qualified for selection had 
they completed the full test battery. Gains of this nature are considerable. 

It is the object, therefore, of this paper to illustrate, streamline, and 
put into more general form this pre-screening technique. 


II. The Objectives of Pre-Screening 
These have been previously (8) defined as follows: 


(i) to reduce the cost of testing by eliminating in the first instance those candidates 
who have little or no chance of obtaining the qualifying mark for selection (a). 

To define quite specifically the operative words “‘little or no chance,”’ 
the basic condition under which pre-screening is permissible as follows: 

The cut-off score (y) on the Pre-Screening Test shall be set in such a way that at that 
level the candidate has a probability of .01 of obtaining or surpassing the Final Battery 
Score (a) at which management has decided selection will ultimately take place. This 
condition can be more concisely stated as, 


ply > a| 7) = Ol. (1) 


This decision would mean in effect that the only candidates who are screened out are those 
whose chances are less than 1 in 100 of making the pre-determined battery score (a) when 
fully tested. It will be demonstrated that this safeguard against discrimination is adequate. 

One further point must be considered in regard to this basic condition. We have 
spoken of the battery score at which management has decided to make the final selection (a). 
This is neither a constant level for all batteries, nor for the same battery on all occasions. 
It will vary according to selection policy (Selection Ratio). This in turn will depend upon 
the balance to be maintained between manning requirements, on the one hand, and training 
wastage, on the other. The point is that for any given test battery these conditions of 
selection will be considered in advance, and the value of a will be known. 

(ii) at the same time pre-screening should ensure that final selection will be no less 
—— than it would have been had all applicants passed through the full testing pro- 
cedures. 

Thus, from management’s point of view we should ensure that no good material is 
being lost in the pre-screening process; and from the candidate’s point of view, that no 
individual is discriminated against by being rejected on the basis of one short test, and 
so prevented from improving his assessment by means of a full testing programme. 

Clearly perfection can never be achieved in the assessment of human material, but 
it will be shown that our basic operative condition in screening, ply >a\|y] = .01, allows 
us to approach very closely to these desired objectives.” 


we shall fix 


III. The Theoretical Model 


Once the above pre-screening policy has been determined, the develop- 
ment of the technique becomes a matter of finding some predictor (x) which 
will correlate highly with the final battery standard score (y). This is usually 
one of the tests contained in the battery itself. It should preferably be a 
pencil-and-paper test which is easy to administer on a group basis. Having 
been included in the final battery score by means of the multiple regression 
equation, it should have high correlation with the latter. If these conditions 
can be satisfied, the psychometric structure of the pre-screening test vis-4-vis 
the final battery score can then be developed in the same way as has been 
done for the final battery score against the criterion. Operating characteristics 
can then be drawn up for the pre-screening test as have been done previously 
for the test battery. The underlying principles and mathematical models are 
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identical and have already been illustrated (2, 7, 8, and 9). These are imme- 
diately applicable to our problem provided the necessary changes are made 
in terms: viz., the point of dichotomy on the predicted variable is now called 
a instead of 6, and the new cut-off point on the predictor variable is y instead 
of a. 

We may thus write Sichel’s (7) formula for the selector’s operating charac- 
teristic as 


ply > «| 2] = of 28], (2) 


x = pre-screening test scores in standard measures, 
y = battery test scores in standard measures, 
ply > a|xz] = the conditional probability of a pre-screening score (x) being 
associated with a score on the battery (y) which is > a pre- 
determined value a, 


where 


and 





a =e lee” de, 


By integrating the product of equation (2) and the normal distribution 





erf (x) —_ Tz oo. 


we get the proportion of the applicant population who are both selected by the 
pre-screening test and make or surpass the battery score a, i.e., 


p= [- o| 58 erf (x) dz. (3) 


From Figure 1 we see that the proportion of applicants who are not selected by 
the pre-screening test but equal or exceed the battery cutting score a is 


= Ba) — p. (4) 


Hence it follows that the proportionate loss of good material for a given set of 
pre-screening conditions is given by 


Number of candidates having battery scores > a lost in pre-screening 





Yin = Total number of candidates in population having battery scores > a 
— — Lf of se] ait 
Pi + Ds ; P(a) ®(a) erf (x) dz. (5) 


We now have the necessary information to construct the mathematical 
model for the pre-screening technique and illustrate how it works in the 
case of a specific test correlating, e.g., .80, .85, or .90 with the full battery. 
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FiaureE 1 
Testing Economy (£) and Loss (L) Achieved by Pre-screening Technique 


The results will be reported in terms of the following: 


(1) The value of y for a given selection ratio and a probability of success of .01 

Without pre-screening, the selection ratio (S.R.) pertaining to the full 

battery is given as @(a). With pre-screening, there is a certain loss y,,, 

defined in (5) which reduces the actual S.R. obtained after the application 

of this technique. To accommodate this loss, the selection ratio resulting 
after pre-screening is precisely defined as 

S.R. = (1 — 41,,)¥(@). (6) 

Equation (6) requires a priori knowledge of a andy. In practice, however, 

it is the selection ratio which is the given quantity. We have to find a and y 


subject to 
ply > a|y] = .01 
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as previously laid down. For this condition we have from equation (2) 


al oe pn | _ O01 


V1— >), 





[2.32635]. 
a — 2.326385 V1 — p (7) 


p 
The solution of equations (6) and (7) for the required a and yo, is not possible 
by elementary means but may be effected with the help of graphs as shown at 
a later stage. 


Hence, 





A a 


(2) The “testing economy’”’ achieved by the pre-screening technique 


Let us define “testing economy” (Z) as the percentage of candidates who, 
with the exception of the pre-screening procedure, need not undergo the 
majority of tests making up the battery. From Figure 1 we deduce 


E = 100[1 — ®(7.0:)]%.- (8) 
Often E is large enough to cause a major saving in testing costs. 


(3) The percentage of potential material lost in pre-screening 


The percentage of candidates who would make or surpass the battery 
selection score a but are lost in the application of the pre-screening technique 
has already partly been defined in equation (5). For the given condition 
ply > aly] = .01, we write the “loss” as 











L = 100¥1,,..:%- (9) 
TABLE 1 
.o , H, and L for Various Selection Ratios* 

(p = .90) 

Selection Battery cutting Pre-screening Testing Loss 
Ratio S.R. Score a cutting score Economy EF L 

(in %) Y.01 (in %) (in %) 
2.23 70 60.96 86.35 1.86 
6.62 65 55.40 70.54 0.94 
15.80 60 49.84 49.36 0.41 
30.80 55 44,29 28.40 0.17 
49.97 50 38.73 12.99 0.06 
69.13 45 33.18 4.63 0.02 





*In this study raw test scores were converted to standard scores having a mean of 50 and standard devia- 
tion of 10. 
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(4) The Calculation of y.., , E, and L for Given Selection Ratios and Correlations 


These quantities were calculated with the help of Arbous’ Tables (9) and 
the above formulas, and are reflected in Table 1 together with the correct 
pre-screening cutting score 7.0: . 

From the above table we see that for a selection ratio of 6.6% and a 
correlation p = .90, the majority of applicants (70.5%) need not be tested 
on the full battery. None of them has a chance exceeding 1 in 100 to be 
selected by the total combination of tests. This major saving in testing 
costs is achieved at a loss of only 0.9% of good material; i.e., for 100 candidates 
who should all be selected by the full battery, only one would be eliminated 
by the pre-screening test. 

The results embodied in Table 1 are graphed in Figures 2, 3, 4, and 5, 
taking the selection ratio as the independent variable. 

For the benefit of those who wish to prepare tables and graphs suiting 
their own needs, we have repeated the necessary calculations based on p = .80 
and .85 as well. The results are shown in Table 2. The corresponding graphs 
based on these values are included also in Figures 2, 3, 4, and 5. 


(5) Appraisal of Results 


In Figure 2 the three curves coincide on the graph for the three levels 
of p = .80, .85 and .90. This is due to the minute loss which is sustained in 
pre-screening. In fact, for most practical purposes, one could disregard this 
loss in deriving values of a, and use directly the tables for the normal integral. 

The most striking feature of Figure 4 is the fact that the differences 
in testing economy (£) between the different values of p are greatest in the 
middle ranges of S.R. which will occur more often in practice. Here a dif- 
ference of .1 in p can have a considerable effect on EZ. E.g., when 8.R. = 20%, 


p= 80, E = 24.0% 
p= 90, E = 42.5%. 


In this particular case the testing economy for p = .90 is nearly twice that of 
p = .80. 

From Figure 5 it will be noted that for large S.R.’s a difference in p 
from .80 to .90 makes virtually no difference in the loss of good material. 
This difference is exceedingly small. It becomes relatively larger, however, 
as S.R. decreases, but in most practical cases (i.e., when S.R. > 7%) the 
losses will be < 1.5% for all values of p > .80. 


IV. An Alternative Model 

In the example given above we have set the operative condition in pre- 
screening as ply > a\y] = .01, and, having done so, indicated to management 
what losses (y,,,.,,) it suffered at the different levels of S.R. 








TABLE 2 
a, y.. , E, and LZ as Functions of Selection Ratio (S.R.) and Correlation Coefficient (p)* 
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*Due to rounding, tabulated values may be in error by one in the last digit. 
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It would have been just as possible to set the condition in management’s 
interest as y,,, = .01, i.e., to permit a loss of 1% of the candidates who would 
make or surpass the pre-determined battery selection score a, under all cir- 
cumstances. It would merely be necessary then in turn to report to the 
candidates at what level of probability of making this score they would have 
been pre-screened out, for different values of S.R., ie., to determine the 
resulting value of ply > aly]. 

This alternative method was fully investigated and was rejected because 
of the fact that for certain values of a which were likely to be met in practice, 
the value of ply > aly] became too high. E.g., when 


a<0.5sm, ply > aly] > .04 
a=0.0s.m, ply > aly] = .08 
in the case of both p = .80 and .90. 


V. The Technique in Practice 


It now becomes necessary to test the mathematical model for the pre- 
screening technique by comparing the expected results with those actually 
obtained in a practical situation. 

In the selection program for potential army recruits a battery of four 
pencil-and-paper tests is administered to the applicants. The intercorrelation 
matrix of the tests based on a sample of 1637 subjects was given in a previous 
study by Sichel (16) as: 








A(G) .637 .596 .690 
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R,.3 = = " (10) 
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i=] j=1 
where 


R,,2 = correlation between 7th test and total battery (including that test), 
battery weight of 7th test, and 
r;; = correlation between ith and jth test, 
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we may find the required correlation between the pre-screening test and 
the full battery. Using A(G) and all 8 = 1/4 we have 
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Ri.8 = 894, 


The correlation based on a scattergram of test A(G) and the complete battery 
scores (both variables in standard scores) checked the above derived correla- 
tion to the last decimal. 

The formulas given above all presuppose a bivariate normal distribution. 
To test this assumption a bivariate normal surface was fitted to the scatter- 
gram using Pearson’s Tables (15). For 36 degrees of freedom and x” = 49.9, 
we found P = .064 so that the hypothesis of binormality cannot be rejected. 

For a given set of a’s advancing from zero in half sigma units correspond- 
ing values of y.o,; were calculated from equation (7). A new scattergram was 
drawn up using the matching a and y.o, values as class boundaries. The 
scattergram is shown in Figure 6. It enables us to compare theory and obser- 
vations by counting the number of applicants who are screened out, being lost 


Test Battery in Standard Scores 
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FiGure 6 
Bivariate Frequency Distribution of Pre-Screening Test Against Full Battery 
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TABLE 3 
Comparison of Theoretical and Actual Values of E, L, and S.R. 














Testing Economy Loss Selection Ratio 
(E%) (L%) (8.R.%) 
@ Y.01 
Expected Observed Expected Observed Expected Observed 

70.0 60.7 85.8 85.9 2.0 2.3 2.2 2.6 
65.0 55.1 69.6 70.2 1.0 2.5 6.6 7.3 
60.0 49.5 48.1 47.8 0.4 0.4 15.8 15.9 
55.0 43.9 27.2 26.5 0.2 0.0 30.8 30.1 
50.0 38.3 12.2 12.8 0.1 0.1 50.0 50.0 
45.0 32.8 4.2 4.6 0.0 0.1 69.1 68.5 





The use of Table 2 is illustrated by the following example: 
Given: R;,3 = .894,S8.R. = 15.8%. Find a, y.9 , E, and L. 


by pre-screening, etc. for matching values of a and y.o,. The results are 
summarized in Table 3. The close agreement between theory and observa- 
tions will be noted. 

The detailed linear interpolations in Table 4 based on Table 2 are self- 
explanatory. The slight discrepancies between the last row of Table 4 and the 
third row of Table 3 are due to rounding errors in the tabular values used. 

The above example may also be used to illustrate the method for obtaining 
the observed values of EF and L from the bivariate distribution in Figure 6, 

In accordance with Figure 1 heavy lines were drawn in Figure 6 vertically 


TABLE 4 


Solution by Interpolation of a, 7,0 , H, and L 
for Illustrative Problem with r = .894 and S.R. = 15.8% 




















p S.R.(%) a Y.01 E% L% 
85 15 60.3 47.8 41.3 0.5 
85 20 58.4 45.6 32.2 0.3 
85 15.8 60.0 47.45 39.84 0.47 
.90 15 60.3 50.2 50.9 0.4 
.90 20 58.4 48.1 42.5 0.3 
.90 15.8 60.0 49.86 49.56 0.38 





49.6 48.4 0.4 
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at a = 60, and horizontally at y = 49.526 forS.R. = 15.8%. The discrepancy 
between the value for y used and that obtained in Table 4 is due to the effects 
of rounding the last decimal in the calculations. The actual value in Table 4 
is 49.57. The theoretical value was 49.526 which we used for the bivariate 
distribution. 
From Figure 6 it will be seen that 
(i) in all, 260 persons make a battery score of a > 60. Of these only 1 is 
lost by being pre-screened out (i.e., having a y score = 49.526). The 
loss (L) is thus 1/260 = 0.4%. 
(ii) 783 persons are pre-screened out and need to do only 1 test. The testing 
economy (£) is thus 783/1637 = 47.8%. 


VI. The Dollar Value of Pre-Screening Economies 


It would have been admirable to express the cost of battery testing and 
pre-screening as functions of some variable, and by some derived equation, to 
have graphed the actual dollar savings of the technique for different testing 
demands and different S.R.’s. 

The fact that this is not possible will be readily appreciated by those 
who have had experience of diverse testing programs. 

It is hoped that the model developed here will enable testers to estimate 
financial savings in their own particular circumstances. 
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THE FACTORIAL COMPOSITION AND VALIDITY 
OF DIFFERENTLY SPEEDED TESTS 


CHARLES T. Myers 


EDUCATIONAL TESTING SERVICE 


Scores on five differently speeded parts of a nonverbal reasoning test 
were analyzed to determine the relationship between the abilities measured by 
the speeded and the unspeeded parts. Two orthogonal factors were found: 
One is described as the ability to answer the problems correctly; the other is 
described as the tendency to answer the problems quickly. The first factor 
was found to be somewhat more valid than the second for predicting grades 
at the U.S. Naval Academy. The suggestion is made that descriptions of tests 
are incomplete unless they include some specification describing the speeded- 
ness of the test. 


Introduction 


The purpose of this study was to compare the validities of short nonverbal 
reasoning tests when different numbers of items were given within the same 
time limits. The criteria for validity were grades and grade averages for the 
first year at the U.S. Naval Academy at Annapolis. Three forms of a figure- 
classification test were given in 1948 to 600 of the midshipmen who were then 
in their first summer at the Academy. The forms were made up of five 
12-minute parts, these parts including either 10, 20, or 30 items with at 
least one part of each form at each speed level. Correlations were computed 
between scores on the different parts, and also between the parts and the 
grades in the first year’s work at the Academy. These intercorrelations 
were analyzed by factorial methods. The following conclusions were drawn 
from this study: that the moderately speeded parts were the most valid and 
that the ability to answer the problems correctly showed no correlation with 
the tendency to answer the problems quickly. 


Description of the Test 


Figure-classification problems of the type used in this study consist of 
two sets of four geometric figures which define the problem, and of five 
multiple-choice answer figures. The persons taking the test are required to 
induce the characteristic possessed by the first four figures but not by the 
second four and to select the only one of the five answer figures which has 
the characteristic common to the first four problem figures. Such charac- 
teristics may involve the size, shape, position, drawing, or number of parts 
of the figures. 
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For this experiment a test of 100 figure-classification items was prepared 
in three different forms. These items were printed on ten pages with ten 
items on a page. The same ten pages were used in each form. In all three 
forms these pages were divided into five 12-minute parts with either one, 
two, or three pages to a part. The three forms differed only in the grouping 
of the pages into these parts of various lengths. The parts of Form A had 
ten, twenty, thirty, ten, and thirty items in that order. The parts of Form B 
had twenty, thirty, ten, ten, and thirty items. The parts of Form C had 
thirty, ten, twenty, ten, and thirty items. The order of the pages and their 
division into parts is shown in Table 1. 


TABLE 1 
Order of Pages and 
Speededness of Parts as Indicated by Per Cent Completing Each Page* 








Form A Form B Form C 
Part 1: 1st Page 97 (a)t 99 (a) 100 (a) 
2nd Page 70 (b) 79 (b) 
3rd Page 41 (d) 
Part 2: Ist Page 98 (c) 99 (c) 91 (c) 
2nd Page 57 (d) 77 (d) 
3rd Page 37 (f) 
Part 3: Ist Page 99 (e) 100 (e) 100 (e) 
2nd Page 65 (f) 65 (f) 
3rd Page 30 (b) 
Part 4: Ist Page 100 (g) 99 (g) 100 (g) 
Part 5: Ist Page 100 (h) 99 (h) 100 (h) 
2nd Page 84 (i) 82 (i) 82 (i) 
3rd Page 36 (j) 34 (j) 38 (j) 





*Per cent of subjects attempting the last item on each page or some subsequent item in the same part. 
tLetters in parentheses identify pages containing identical sets of ten items. 


This arrangement made it possible to observe whether practice on the 
test would change the effect of speededness. Since each different speed level 
was used as part one of some form and since in each form one of the last two 
parts was unspeeded and the other highly speeded, it was possible to compare 
each speed level when given without any practice, with the extreme speed 
levels after 36 minutes and 60 items of practice. 


Administration of the Test 


The test was administered to a group of approximately 600 midshipmen 
during their first summer at the U.S. Naval Academy.* The three forms of 


*Grateful acknowledgement is made to the officers of the Naval Academy for their 
assistance in administering the test and for providing the data for the validation of the test. 
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the test were distributed in rotation according to the order in which the men 
happened to be seated, each man taking only one form. All three forms were 
administered simultaneously. The directions on the cover page included the 
following statement: ‘‘Not all the parts have the same number of problems; 
the number is printed at the beginning of each part.” The cover page also 
indicated that twelve minutes were to be allowed for each part. 


Test and Criterion Scores 


Nineteen scores were obtained for each form. ‘These scores were as 
follows: For each of the ten pages the number of problems correct; for each 
two-page part the number of problems correct; and for each three-page part 
the number of problems correct, the number of problems wrong, the number 
skipped, and the number attempted. By “number skipped” is meant the 
number of unanswered problems preceding the last problem that was an- 
swered; by “number attempted” is meant the number of the last problem 
that was answered. 

After the academic year which followed the administration of the test 
the Naval Academy furnished course grades of the men for that year in the 
form of rank-order scores. The men were graded on the following: Conduct, 
Physical Training, Marine Engineering, Mathematics, Electrical Engineering, 
English, Foreign Languages, and a Relative Standing which is a weighted 
average of all the grades. These rank-order scores were converted to normal 
distributions with means of 50 and standard deviations of 10 for the whole 
class. 


Speededness of the Parts 


The speededness of the parts of the test may be judged by the percentages 
of persons marking responses at various points through the test. This is also 
shown in Table 1. The most speeded parts were finished by from 30 to 41 
per cent of the subjects. In four-fifths of all the parts the tenth problem was 
attempted by ninety-nine per cent of the subjects. None of the parts was a 
“pure speed test’ since they all contained some difficult items. For the 
30-item parts the variance of the number of items wrong was about equal to 
the variance of the number of items attempted. 


Analysis of the Intercorrelations 


Product-moment correlations between all the test and criterion scores 
were obtained by I. B. M. machine methods. The three tables of intercorre- 
lations were analyzed by Thurstone’s grouping method. This was done to 
help evaluate the effect of the time limits on the validity of the test. Scores 
on the first page of each of the five parts were used as a group to define the 
first factor. Scores on the last page of each of the two three-page parts 
together with the scores on the number of items attempted on the same two 
parts were used as a group to define the second factor. The two groupings 
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were selected because the scores in each group had high intercorrelations 
among themselves and low intercorrelations with other scores, and because 
the proportionality of the rows of intercorrelations suggested that their vectors 
would form clusters. After the extraction of the second factor the small size of 
the residuals indicated that no more factors could be extracted from the page- 
score correlations, and the correlations between test scores and the criterion 
scores were mostly accounted for by these two factors. For the three matrices 
a distribution was made of these residuals after the second factor had been 
extracted. This distribution did not include the residuals of the intercorrela- 
tions between criterion scores or between the wrong, skip, or number-at- 
tempted scores. None of the 546 residuals in this distribution had an absolute 
value of more than .17 and only 28 of the residuals had an absolute value of 
more than .10. Over two thirds of the residuals were within the range of 
.05 to —.05. 

The complete matrices included a few cells expressing relationships be- 
tween experimentally dependent scores. In these cells values were substituted 
from other parts of the same matrix. For each cell expressing the relationship 
of dependent scores there were two other cells, one in the same row and the 
other in the same column, expressing the relationship of those two scores with 
similar but independent measures. The substitute value that was used was 
the arithmetic mean of those two cell entries. Preliminary investigation using 
five independent variables, three for the unspeeded and two for the speeded 
parts, indicated that the above procedure would not distort the configuration 
of vectors. 

The two factors which were found in each of the matrices were able to 
account for practically all the correlation between the test scores as well as 
for the validity of the tests. Factor loadings are shown in Table 2. The first 
factor that was extracted was defined by the scores of the number right on the 
first pages of all the parts, of whatever length, and may be described as the 
ability to answer the problems correctly. The second factor was defined by 
either the number of problems answered correctly on the last pages of the 
30-problem parts, or the number of problems attempted in those parts. 
This factor may be described as the tendency to answer the problems quickly. 

All three analyses resulted in similar configurations of test and criterion 
vectors. The fifteen scores used to define the first factor in the three analyses 
had loadings on that factor in the range from .51 to .75. The use of the group- 
ing method without rotation necessarily resulted in some positive and some 
negative loadings on the second factor for these vectors in each analysis. 
The homogeneity of these scores can be judged from the observation that 
their loadings on the second factor fell within the range .08 to —.10. In this 
method the factors must be orthogonal, but the loadings of the tests defining 
the second factor may have any value on the first factor. However, in these 
configurations the scores used to define the second factor had loadings in the 
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range .04 to —.19 on the first factor with a median of —.10 whereas their 
loadings on the second factor were in the range .55 to .81. It is concluded 
that the two groups of scores used to define the factors were each homogeneous 
and that the two groups were orthogonal. 

Inspection of the factor loadings and of the table of percentages of persons 
attempting the last item on each page shows what appears to be a slight 
systematic effect of practice; but this effect is negligible and could not in- 
fluence the conclusions regarding the effects of speededness. 


TABLE 2 
Factor Loadings 











Form A Form B Form C 
I II I II I II 

Page 1 {[ .69 Oi" [ .60 .08* [ o@ .05* 
Page 2 [ 58 .03* 20 .50 | 32 .55 
Page 3 21 .25 66 s01* 03 .57t 
Page 4 [ 68 —.04* 56 .38 [ 66 —.07* 
Page 5 49 .23 tL 04 627 cr ft = OL 
Page 6 L—.08 .66t [ .72 —.05* L 40.87 
Page 7 [ 66 —.05* [ 59 —.10* [ 62 .08* 
Page 8 f  .57 .04* 62 .06* [ 51 —.05* 
Page 9 | .60 .13 54 .34 46 .37 
Page 10 L—.10 .63t ~.05 .55t L—.10 57} 
Wrong —.51 .45 — .46 46 —.55 .39 
Skipped —.05 04 —.02 —.03 13 —.03 
Attempted —.14 .69F — .08 .72T —.13 .75T 
Wrong — .52 .48 — .58 .47 — .58 .39 
Skipped —.06 —.03 .06 —.03 .16 —.03 
Attempted —.19 fot —.15 .81f —.17 By i iy 3 
Conduct .00 05 Bile .05 .09 .22 
Physical Training .05 —.04 .07 13 —.01 —.02 
Marine Engineering .42 .14 .36 14 .35 . 28 
Mathematics .36 11 .26 .08 .24 .15 
Electrical Engin. 32 .06 21 .00 .28 .14 
English .09 12 22 .02 Se 05 
Foreign Languages .09 .06 24 .06 21 —.07 
Relative Standing 35 .09 .o2 .10 .o2 .14 
Part Scores on Test 

(2) .47 21 (1) .51 .39 (1) .48 57 

(3) .52 .38 (2) .57 .53 (3) .67 .23 

(5) .58 38 (5) .55 .49 (5) .50 .55 





Note that page numbers indicate the order and not the identity of the pages (which may be determined from 
Table 1) and that these pages occupy different positions in the various parts of each form. Brackets indicate 
the extent of the separately timed parts. 

*Scores used in the group defining the first factor. 
+Scores used in the group defining the second factor. 
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The criterion scores for Conduct and Physical Training had low and 
inconsistent loadings on both factors. The loadings of the other criterion 
scores on the first factor had a median of .27 and were in the range .09 to .42. 
These same scores had loadings in the second factor in the range —.07 to .28 
with a median at .10. The part-score vector in each of the three analyses 
which lay closest to the median value for the criterion scores was the vector 
for the 20-item part. This indicates the superior validity of the moderately 
speeded test. As was expected, the 30-item parts all had higher loadings on 
the second factor than did the 20-item parts, and the 10-iiem parts all had 
lower loadings than did the 20-item parts. 


Conclusions 


Two sets of scores on this test form orthogonal clusters. Scores on the 
first pages of all parts have significant loadings on only the first factor. 
Scores on the last pages of speeded parts have significant loadings on only the 
second factor. The criterion scores, grades at the U. S. Naval Academy, 
are more closely related to scores on first pages than to scores on last pages. 
The conclusion of this study is that the most valid figure-classification test 
for predicting grades at the Naval Academy is a moderately speeded test 
which can be finished by about 70% of the candidates. The score on speeded 
tests is a function of two orthogonal factors, the factor of ability being more 
valid than the rate-of-answering factor. 

Ordinarily one assumes that those who are able to solve difficult problems 
in a test will work more quickly than those who cannot. This study suggests 
that this may not always be the case. The hypothesis is advanced that for 
some speeded tests not every mark on the answer sheet represents a subject’s 
reasoned conception of an adequate solution. Many of the answers may be 
guesses, although not necessarily random answers. The group of subjects 
who reach the end of a speeded test may include some who can solve the 
problems quickly and also some who answer the problems before they complete 
the solutions. These subjects may not understand what they are expected to 
do or they may prefer to guess in order to work quickly. Thus, while it may 
not always be possible to describe the full effects of speededness by stating 
the proportion of answers marked, nevertheless the evaluation of a test cannot 
be complete without some knowledge of the speed at which the subjects were 
required to work.* 

*For more complete and detailed information, including tables of intercorrelations and 
centroid analyses of one of these forms and of an additional form, see Myers, Charles T. 


The factorial composition and validity of differently speeded tests (second edition) Research 
Bulletin #52-1, Educational Testing Service, Princeton, New Jersey, 1952. 
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BOOK REVIEW 


Paut S. Dwyer. Linear Computations. New York: John Wiley and Sons, Inc., 1951. 
Pp. xi + 344. 


Linear Computations is addressed to the theoretical and practical considerations 
of the general problem of finding numerical solutions for systems of linear equations. 
The author hoped that this volume might serve as a textbook for a course in linear com- 
putations and also as a reference book—and from all appearances it seems that both 
these objectives will be attained. The organization of the material follows sound pedagogical 
principles which, with the many illustrative problems and the exercises at the close of each 
chapter, make it especially suitable as a textbook. At the same time, the orderly grouping 
of methods, the illustrative problems, the annotated references at the end of each chapter, 
and the detailed index will be appreciated by the research worker who uses it as a reference 
work in computing techniques. 

It would be very presumptuous of the reviewer to attempt to summarize the breadth 
of the theory and technique covered in Lincar Computations in a few short pages. Instead, 
he can but hope that a brief outline of its contents will indicate the scope of the volume. 

Introductory remarks, defining and delimiting the problem of linear computations, 
and general principles of computational design are covered in Chapters 1 and 3. Special 
emphasis is given in this book (Chapters 2 and 17) to the question of errors in the solutions 
of simultaneous linear equations when the coefficients themselves are recognized as being 
subject to error. While some of the classical material on calculations with significant 
numbers is presented, what is much more important is the introduction of “range numbers” 
or “approximation-error numbers” for precise results. However, extensive computations 
with these numbers would be required in determining bounds for the error in the basic 
linear problems. Therefore, the author recommends the use of “incomplete numbers” 
with separate estimates of the maximum errors in linear computations such as evaluation 
of determinants, solutions of simultaneous equations, and determination of elements of the 
inverse matrix. 

An orderly grouping of various methods for solving systems of linear equations is 
carried out in Chapters 4, 5, and 6, and the relationships among these methods are brought 
out in Chapter 7. Then applications are made (Chapter 8) to the solution of many sets of 
simultaneous linear equations derived from the original] set by deletion of certain variables 
or by replacement of the constant terms by new values. The different computational 
methods are exhibited by schematic arrangement, in general algebraic form, and then 
followed by numerical illustrations to assist the reader in understanding the development. 

Next, an elementary exposition of the theory of determinants is presented (Chapter 
9), followed by the application of the various reduction methods to the evaluation of 
determinants (Chapter 10). Then the reader is introduced to the basic algebra of matrices 
in Chapter 12. Matrix theory enables the author to develop a more powerful exposition 
of the solution of simultaneous linear equations, employing the calculation of the inverse 
matrix (Chapters 13 and 14). Also, certain problems involving the characteristic equation 
are solved by the method of determinants (Chapter 15). The Hotelling-Bingham-Girshick 
method is presented for obtaining the inverse of a matrix based on the characteristic 
equation; and an excellent computational modification of the recently developed recursion 
formula of J. S. Frame is presented for the determination of the adjoint, the determinant, 
and the inverse as well as the characteristic equation. 

Because of the many diverse approaches to the problem of solving simultaneous 
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linear equations (and related problems), the author frankly admits that it would be an 
impossible task to review all of them. He devotes Chapter 16 to the treatment of other 
methods (e.g., extension and iterative methods) which he does not recommend for use 
with desk calculators. 

While the general] problem of finding numerical solutions to systems of linear equa- 
tions arises in a variety of ways, least squares theory and related statistical studies immedi- 
ately lead to such mathematical problems. The presentation in this book is of a general 
mathematical nature rather than in specific statistical terms, although problems in statistics 
are mentioned. Then in Chapter 18 the author exhibits a number of statistical applications 
to assist the reader in translating the mathematical results to appropriate statistical 
results, with emphasis on improvement of the computational design. 

Finally, in the concluding chapter, the author looks ahead to procedures for generaliz- 
ing the techniques presented to non-linear problems; and points to the continued major 
role that the desk calculator will play (high-speed digital computers notwithstanding) in 
solving linear problems involving 15 to 20 variables by the elimination methods featured 
in this book. 

The general organization of the text, designation of sections and tables, and over-all 
typography is such as to enhance the usefulness of this book as a reference work. However, 
in a volume of this type (with such extensive numerical and algebraic illustrations) it is 
almost inevitable that errors will appear in the initial printing. A list of such corrections 
has been prepared by the author and will be distributed by the publishers. 

Linear Computations is the outgrowth of Professor Dwyer’s many years of experience 
and interest in both the development of complex theory in mathematical statistics and 
the design of appropriate computational techniques for the application of such theory. 
The practicability of the book was assured by the author’s years of experience in applying 
the methods to problems in the Statistical Research Laboratory at the University of 
Michigan, and more recently as a consultant on such matters to the Department of the 
Army. The use of Linear Computations will extend to many fields. It should provide an 
excellent foundation in numerical analysis—a subject of increasing importance and interest 
since the rapid development of high-speed electronic digital computers. Psychological 
statisticians and research psychologists, in particular, will weleome the many computa- 
tional techniques applicable to their special problems. However, the important implications 
will not be arrived at by scanning, but diligent work with the material presented can 
produce very profitable results in psychometrics. 


Personnel Research Section, Harry H. Harman 
Adjutant General’s Office 
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Three new issues of the Psychometric Monograph Series are in press. 
Nos. 6, 7, and 8 will be ready for distribution this fall. 
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Psychometric Monograph No. 5, $4.00. 
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Lord, Frederic. A theory of test scores. 
Psychometric Monograph No. 7, $2.00. 


Roff, Merrill. A factorial study of tests in the perceptual area. 
Psychometric Monograph No. 8, $1.50. 


Orders for Psychometric Monographs No. 1 through No. 6 should be 
sent to The University of Chicago Press, 5750 Ellis Avenue, Chicago 37, 
Illinois until December 31, 1952. All orders for later issues and all orders for 
No. 1 through No. 6 beginning January 1, 1953 should be sent to The William 
Byrd Press, Box 2-w, Richmond 5, Virginia. 

















